Cisco 9200 - counter based check on interface didn't work

Since we installed the Cisco 9200 switch models a few months ago, parts of the interface information have not been read out correctly.

The speed and up / down status are displayed correctly, but all counter based reports no longer work.

The service metrics consistently show only 0.

Input Discards: 0/s
Input Errors: 0/s
Input bandwidth: 0.00 bit/s
Input non-unicast packets: 0/s
Input unicast packets: 0/s
Length of output queue: 0
Output Discards: 0/s
Output Errors: 0/s
Output bandwidth: 0.00 bit/s
Output non-unicast packets: 0/s
Output unicast packets: 0/s

The check outputs show the following:

in=0;;;0;12500000 inucast=0;;;; innucast=0;;;; indisc=0;;;; inerr=0;0.01;0.1;; out=0;;;0;12500000 outucast=0;;;; outnucast=0;;;; outdisc=0;;;; outerr=0;0.01;0.1;; outqlen=0;;;;

OK - [GigabitEthernet1/0/12] (up) MAC: 10:B3:D6:12:FD:8C, 100 Mbit/s, In: 0 B/s (0.0%), Out: 0 B/s (0.0%)

The check plugin “check_mk-if64” is used.

If you manually query the switch via snmpwalk, the traffic counter is displayed correctly.

IF-MIB::ifInUcastPkts.19 = Counter32: 13735079
IF-MIB::ifInUcastPkts.20 = Counter32: 20575863
IF-MIB::ifInUcastPkts.21 = Counter32: 12424430

IF-MIB::ifInOctets.19 = Counter32: 1588366609
IF-MIB::ifInOctets.20 = Counter32: 2950329060
IF-MIB::ifInOctets.21 = Counter32: 1367865459

IF-MIB::ifOutOctets.19 = Counter32: 2103320498
IF-MIB::ifOutOctets.20 = Counter32: 3440059124
IF-MIB::ifOutOctets.21 = Counter32: 1889509885

With all other cisco switch models, the counter-based displays work without problems.

I installed the checkmk version 6p13.

I have already deleted the host and created a new one, that didn’t work.

Firmware 16.12.1 is currently running on the switches.

Does anyone have any idea what this could be and how to get the problem solved?

This plugin is querying not the mentioned 32 bit counter but the high counters.
The following Table shows all the OIDs used by if64.

.1.3.6.1.2.1.2.2.1.1	        ifIndex                   0
.1.3.6.1.2.1.2.2.1.2	        ifDescr                   1
.1.3.6.1.2.1.2.2.1.3	        ifType                    2
.1.3.6.1.2.1.2.2.1.5	        ifSpeed                   3
.1.3.6.1.2.1.2.2.1.8	        ifOperStatus              4
.1.3.6.1.2.1.31.1.1.1.6	        ifHCInOctets              5
.1.3.6.1.2.1.31.1.1.1.7	        ifHCInUcastPkts           6
.1.3.6.1.2.1.31.1.1.1.8	        ifHCInMulticastPkts       7
.1.3.6.1.2.1.31.1.1.1.9	        ifHCInBroadcastPkts       8
.1.3.6.1.2.1.2.2.1.13	        ifInDiscards              9
.1.3.6.1.2.1.2.2.1.14	        ifInErrors               10
.1.3.6.1.2.1.31.1.1.1.10        ifHCOutOctets            11
.1.3.6.1.2.1.31.1.1.1.11        ifHCOutUcastPkts         12
.1.3.6.1.2.1.31.1.1.1.12        ifHCOutMulticastPkts     13
.1.3.6.1.2.1.31.1.1.1.13        ifHCOutBroadcastPkts     14
.1.3.6.1.2.1.2.2.1.19           ifOutDiscards            15
.1.3.6.1.2.1.2.2.1.20           ifOutErrors              16
.1.3.6.1.2.1.2.2.1.21           ifOutQLen                17
.1.3.6.1.2.1.31.1.1.1.18        ifAlias                  18
.1.3.6.1.2.1.31.1.1.1.15        ifHighSpeed              -1  (parse_if64 assumes this is the last element)

If you get no output on these OIDs on your switch I would contact my vendor and request an exchange as the device looks broken.

But before we do such things. How does your SNMP config on the switch looks like?
Has the used SNMP user/account the correct rights and can this user view all the complete OID tree?

The snmp config basically works. If you use snmpwalk with the same snmp-version-2c community as is used in checkmk, the corresponding interface counter will be returned.

sh snmp

Chassis: J#########A
42081047 SNMP packets input
    0 Bad SNMP version errors
    0 Unknown community name
    0 Illegal operation for community name supplied
    0 Encoding errors
    452022 Number of requested variables
    0 Number of altered variables
    438121 Get-request PDUs
    13901 Get-next PDUs
    0 Set-request PDUs
    0 Input queue packet drops (Maximum queue size 1000)
42081047 SNMP packets output
    0 Too big errors (Maximum packet size 1500)
    0 No such name errors
    0 Bad values errors
    0 General errors
    42081047 Response PDUs
    0 Trap PDUs
Packets currently in SNMP process input queue: 0
SNMP global trap: disabled

SNMP logging: disabled

The snmp config:

snmp-server community snmpwrite RW
snmp-server community snmpread RO

We have the problem with six different cisco 9200s. Probably not all devices are faulty.

Is there a log that shows how the data obtained via snmp is transferred to the checkmk database?

Do you checked the OIDs i posted?

Yes, the switch responds to the OIDs with meaningful data. The counters also go up over time.

OMD[###]:~$ snmpwalk -v 2c -c snmpread 172.19.1.105 .1.3.6.1.2.1.31.1.1.1.7
IF-MIB::ifHCInUcastPkts.3 = Counter64: 0
IF-MIB::ifHCInUcastPkts.9 = Counter64: 2
IF-MIB::ifHCInUcastPkts.10 = Counter64: 349162
IF-MIB::ifHCInUcastPkts.11 = Counter64: 14080966
IF-MIB::ifHCInUcastPkts.12 = Counter64: 2
IF-MIB::ifHCInUcastPkts.13 = Counter64: 2
IF-MIB::ifHCInUcastPkts.14 = Counter64: 1747720
IF-MIB::ifHCInUcastPkts.15 = Counter64: 2
IF-MIB::ifHCInUcastPkts.16 = Counter64: 30309
IF-MIB::ifHCInUcastPkts.17 = Counter64: 2
IF-MIB::ifHCInUcastPkts.18 = Counter64: 140575
IF-MIB::ifHCInUcastPkts.19 = Counter64: 13735079
IF-MIB::ifHCInUcastPkts.20 = Counter64: 20575863
IF-MIB::ifHCInUcastPkts.21 = Counter64: 12424430
IF-MIB::ifHCInUcastPkts.22 = Counter64: 12262332
IF-MIB::ifHCInUcastPkts.23 = Counter64: 2
IF-MIB::ifHCInUcastPkts.24 = Counter64: 2
IF-MIB::ifHCInUcastPkts.25 = Counter64: 2
IF-MIB::ifHCInUcastPkts.26 = Counter64: 2
IF-MIB::ifHCInUcastPkts.27 = Counter64: 2
IF-MIB::ifHCInUcastPkts.28 = Counter64: 2
IF-MIB::ifHCInUcastPkts.29 = Counter64: 2
IF-MIB::ifHCInUcastPkts.30 = Counter64: 28409899
IF-MIB::ifHCInUcastPkts.31 = Counter64: 2
IF-MIB::ifHCInUcastPkts.32 = Counter64: 30373736
IF-MIB::ifHCInUcastPkts.33 = Counter64: 2
IF-MIB::ifHCInUcastPkts.34 = Counter64: 9758291
IF-MIB::ifHCInUcastPkts.35 = Counter64: 2
IF-MIB::ifHCInUcastPkts.36 = Counter64: 2
IF-MIB::ifHCInUcastPkts.37 = Counter64: 2
IF-MIB::ifHCInUcastPkts.38 = Counter64: 365651
IF-MIB::ifHCInUcastPkts.39 = Counter64: 2
IF-MIB::ifHCInUcastPkts.40 = Counter64: 2072748
IF-MIB::ifHCInUcastPkts.41 = Counter64: 2
IF-MIB::ifHCInUcastPkts.42 = Counter64: 2
IF-MIB::ifHCInUcastPkts.43 = Counter64: 2
IF-MIB::ifHCInUcastPkts.44 = Counter64: 2
IF-MIB::ifHCInUcastPkts.45 = Counter64: 2
IF-MIB::ifHCInUcastPkts.46 = Counter64: 2
IF-MIB::ifHCInUcastPkts.47 = Counter64: 2
IF-MIB::ifHCInUcastPkts.48 = Counter64: 6877031
IF-MIB::ifHCInUcastPkts.49 = Counter64: 2
IF-MIB::ifHCInUcastPkts.50 = Counter64: 19494848
IF-MIB::ifHCInUcastPkts.51 = Counter64: 2
IF-MIB::ifHCInUcastPkts.52 = Counter64: 5087228
IF-MIB::ifHCInUcastPkts.53 = Counter64: 2
IF-MIB::ifHCInUcastPkts.54 = Counter64: 7149145
IF-MIB::ifHCInUcastPkts.55 = Counter64: 15884619
IF-MIB::ifHCInUcastPkts.56 = Counter64: 3479274
IF-MIB::ifHCInUcastPkts.61 = Counter64: 176255606
IF-MIB::ifHCInUcastPkts.62 = Counter64: 49345149
IF-MIB::ifHCInUcastPkts.63 = Counter64: 0
IF-MIB::ifHCInUcastPkts.64 = Counter64: 0
IF-MIB::ifHCInUcastPkts.68 = Counter64: 3159380


OMD[###]:~$ snmpwalk -v 2c -c snmpread 172.19.1.105 .1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.9 = Counter64: 136
IF-MIB::ifHCInOctets.10 = Counter64: 64585949
IF-MIB::ifHCInOctets.11 = Counter64: 1081228234
IF-MIB::ifHCInOctets.12 = Counter64: 136
IF-MIB::ifHCInOctets.13 = Counter64: 136
IF-MIB::ifHCInOctets.14 = Counter64: 185703687
IF-MIB::ifHCInOctets.15 = Counter64: 136
IF-MIB::ifHCInOctets.16 = Counter64: 31942555
IF-MIB::ifHCInOctets.17 = Counter64: 136
IF-MIB::ifHCInOctets.18 = Counter64: 51759598
IF-MIB::ifHCInOctets.19 = Counter64: 1588366609
IF-MIB::ifHCInOctets.20 = Counter64: 2950329060
IF-MIB::ifHCInOctets.21 = Counter64: 1367865459
IF-MIB::ifHCInOctets.22 = Counter64: 1227860204
IF-MIB::ifHCInOctets.23 = Counter64: 136
IF-MIB::ifHCInOctets.24 = Counter64: 136
IF-MIB::ifHCInOctets.25 = Counter64: 136
IF-MIB::ifHCInOctets.26 = Counter64: 136
IF-MIB::ifHCInOctets.27 = Counter64: 136
IF-MIB::ifHCInOctets.28 = Counter64: 136
IF-MIB::ifHCInOctets.29 = Counter64: 136
IF-MIB::ifHCInOctets.30 = Counter64: 2682802165
IF-MIB::ifHCInOctets.31 = Counter64: 136
IF-MIB::ifHCInOctets.32 = Counter64: 3523151151
IF-MIB::ifHCInOctets.33 = Counter64: 136
IF-MIB::ifHCInOctets.34 = Counter64: 1232567473
IF-MIB::ifHCInOctets.35 = Counter64: 136
IF-MIB::ifHCInOctets.36 = Counter64: 136
IF-MIB::ifHCInOctets.37 = Counter64: 136
IF-MIB::ifHCInOctets.38 = Counter64: 214099206
IF-MIB::ifHCInOctets.39 = Counter64: 136
IF-MIB::ifHCInOctets.40 = Counter64: 228343213
IF-MIB::ifHCInOctets.41 = Counter64: 136
IF-MIB::ifHCInOctets.42 = Counter64: 136
IF-MIB::ifHCInOctets.43 = Counter64: 136
IF-MIB::ifHCInOctets.44 = Counter64: 136
IF-MIB::ifHCInOctets.45 = Counter64: 136
IF-MIB::ifHCInOctets.46 = Counter64: 136
IF-MIB::ifHCInOctets.47 = Counter64: 136
IF-MIB::ifHCInOctets.48 = Counter64: 1539078450
IF-MIB::ifHCInOctets.49 = Counter64: 136
IF-MIB::ifHCInOctets.50 = Counter64: 4530497203
IF-MIB::ifHCInOctets.51 = Counter64: 136
IF-MIB::ifHCInOctets.52 = Counter64: 1161292174
IF-MIB::ifHCInOctets.53 = Counter64: 136
IF-MIB::ifHCInOctets.54 = Counter64: 1468581330
IF-MIB::ifHCInOctets.55 = Counter64: 1612762706
IF-MIB::ifHCInOctets.56 = Counter64: 759925069
IF-MIB::ifHCInOctets.61 = Counter64: 76595704672
IF-MIB::ifHCInOctets.62 = Counter64: 43859179880
IF-MIB::ifHCInOctets.63 = Counter64: 0
IF-MIB::ifHCInOctets.64 = Counter64: 39244
IF-MIB::ifHCInOctets.68 = Counter64: 208218314

Then i don’t know why your checks don’t show anything.

The problem is with the firmware of the switches. I was obviously wrong about the rising counters.

See the following link: https://quickview.cloudapps.cisco.com/quickview/bug/CSCvt22238

May 10, 2020
Products (1)
Cisco Catalyst 9200 Series Switches
Known Affected Releases
16.12.1 16.12.1s 16.12.2 16.12.3 17.1.1 17.2.1
Description (partial)
Symptom:
Below symptoms can be observed:

1. Interface counters do not increment but traffic is forwarded
2. MAC learning issue : new MACs will not be learned, existing MACs may not be aged out, and MAC moves wont be updated. 
3. Packet drops to specific hosts in subnet

Conditions:
Catalyst 9200L/9200 platforms running 16.12.1 or 16.12.2 or 16.12.3 or 17.1.1 or 17.2.1 versions after 49.7 days of uptime.

After the update to 16.12.3a the display works properly again.

Thanks for the support!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.