Untracable network interface errors on FreeBSD / pfSense

[hello]

This happens on both 2.0 and 1.6.0p21:

CRIT	Interface igb1	CRIT - [2] on hostname: (up) MAC: 0C:C4:7A:AA:EE:FD, 1 Gbit/s, In: 138.66 kB/s (0.1%), in-errors: 0.36% CRIT >= 0.1%,    Out: 143.83 kB/s (0.1%)	10.1 s	10.1 s	1.14 Mbit/s / 1.18 Mbit/s
WARN	Interface igb2	WARN - [3] on hostname: (up) MAC: 0C:C4:7A:AA:EE:FD, 1 Gbit/s, In: 130.15 kB/s (0.1%), in-errors: 0.05% WARN >= 0.01%,   Out: 131.13 kB/s (0.1%)	70 s	10.1 s	1.07 Mbit/s / 1.07 Mbit/s
CRIT	Interface lagg0	CRIT - [9] on hostname: (up) MAC: 0C:C4:7A:AA:EE:FD, 2 Gbit/s, In: 268.81 kB/s (0.1%), in-errors: 0.22% CRIT >= 0.1%,    Out: 274.95 kB/s (0.1%)	10.1 s	10.1 s	2.20 Mbit/s / 2.25 Mbit/s

I cannot find any errors in pfsense, in any logs, there are no counters going up, there is no trace of these that I can find. We also monitor SNMP and IPMI and nothing reports any errors.

I suspect there’s a bug somewhere monitoring LAGG (bond, link aggregation).

How can I troubleshoot this and find out whether I can safely up the tresholds / ignore these?

Thanks for the help!

[tosch (Tobias Schönau)]

Hi @hello,

this are errors in your network checkmk just reports. This is not a bug but a situation you need to analyse in your network and the devices included. In-Error means your device is getting TCP packages which are corrupted and thrown away and are resent by the source. This is mostly a sign for a deeper network problem, a broken cable, a dirty fiber optic cable, a broken network card or just buggy firmware.

[hello]

Thanks for the reply.

The point is, pfsense (which collects a lot of stats of itself) is not reporting any errors that I can see. Neither is the switch everything is connected to. If checkmk is right, why isn’t it visible elsewhere?

As you may have noticed, the errors only happen on igb1 and igb2 which are part of an aggregated link (lacp bond). The WAN interface which is a single interface doesn’t report any errors. Same drivers, same network interface hardware, same switch.

Furthermore, lagg0 is only used as a host for about 50 vlan interfaces. From the checkmk errors I cannot see what the source is so I think its virtually impossible to figure this out by doing catch-all packet captures.

Any suggestions?

[tosch (Tobias Schönau)]

I am not a network admin and i don’t know the devices and tools you are mentioning.
I am just explaining the meaning of your post. Checkmk just gathers this information via SNMP (or via another way) and reports it, there is no magic inside checkmk where this information comes from. So in conclusion your device have to know about otherwise it won’t report this errors.
May take a look at the SNMP (or raw) data of your device and you will found this values there.

You may could find some further information if you take a look on the devices connected to this ports, any of this devices should have the corresponding counter parts as out-errors on the network interface.

[hello]

Thanks, I appreciate your time and effort to help out.

I finally found out what checkmk is looking at (by searching freebsd forums instead of pfsense):

Name   Mtu  Network       Address           Ipkts      Ierrs        Idrop Opkts      Oerrs Coll
igb1   9000 <Link#2>      0c:c4:7a:aa:ee:fd 1295088661  8385116     0     2079622938    0  0
igb2   9000 <Link#3>      0c:c4:7a:aa:ee:fd 1012374631  8324729     0     1911317312    0  0
lagg0  9000 <Link#9>      0c:c4:7a:aa:ee:fd 2307463292 16709845     0     3990940250 6995  0