Weird spikes on interfaces

Troubleshooting
1

CMK version: Raw 2.1.0p25
OS version:Rocky 8

Error message:No error message that i know of

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.1.0p25
Try license usage history update.
Trying to acquire lock on /omd/sites/***/var/check_mk/license_usage/next_run
Got lock on /omd/sites/***/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/***/var/check_mk/license_usage/history.json
Got lock on /omd/sites/***/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/***/var/check_mk/license_usage/history.json
Released lock on /omd/sites/***/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/***/var/check_mk/license_usage/next_run
Released lock on /omd/sites/***/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.SNMP
[cpu_tracking] Start [7ff92ca4e430]
[SNMPFetcher] Fetch with cache settings: SNMPFileCache(accsw04.***.is, base_path=/omd/sites/***/tmp/check_mk/data_source_cache/snmp, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 4 sec, allowed is 0 sec)
[SNMPFetcher] Execute data source
  SNMP scan:
       Getting OID .1.3.6.1.2.1.1.1.0: Running 'snmpget -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.2.1.1.1.0'
SNMP answer: ==> ["Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)]
b'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Getting OID .1.3.6.1.2.1.1.2.0: Running 'snmpget -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.2.1.1.2.0'
SNMP answer: ==> [.1.3.6.1.4.1.9.1.616]
b'.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Getting OID .1.3.6.1.4.1.232.2.2.4.2.0: Running 'snmpget -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.4.1.232.2.2.4.2.0'
SNMP answer: ==> [No Such Object available on this agent at this OID]
failed.
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Getting OID .1.3.6.1.2.1.2.2.1.*: Running 'snmpgetnext -Cf -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1'
SNMP answer: ==> [1]
b'1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Getting OID .1.3.6.1.2.1.47.1.1.1.1.*: Running 'snmpgetnext -Cf -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1'
SNMP answer: ==> ["WS-C3560G-48PS"]
b'WS-C3560G-48PS'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Getting OID .1.3.6.1.4.1.674.*: Running 'snmpgetnext -Cf -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.4.1.674'
SNMP answer: ==> [0]
failed.
       Getting OID .1.3.6.1.4.1.2036.2.1.1.4.0: Running 'snmpget -v2c -c *** -m "" -M "" -t 15.00 -r 4 -On -OQ -Oe -Ot 10.10.15.244 .1.3.6.1.4.1.2036.2.1.1.4.0'
SNMP answer: ==> [No Such Object available on this agent at this OID]
failed.
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.2.0: '.1.3.6.1.4.1.9.1.616'
       Using cached OID .1.3.6.1.2.1.1.1.0: 'Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1'
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
       Using cached OID .1.3.6.1.4.1.232.2.2.4.2.0: None
   SNMP scan found                    hr_mem inv_cisco_vlans inv_if snmp_extended_info snmp_os snmp_uptime
Trying to acquire lock on /omd/sites/***/tmp/check_mk/snmp_scan_cache/accsw04.***.is.10.10.15.244
Got lock on /omd/sites/***/tmp/check_mk/snmp_scan_cache/accsw04.***.is.10.10.15.244
Releasing lock on /omd/sites/***/tmp/check_mk/snmp_scan_cache/accsw04.***.is.10.10.15.244
Released lock on /omd/sites/***/tmp/check_mk/snmp_scan_cache/accsw04.***.is.10.10.15.244
hr_mem: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.25.2.3.1.2'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.25.2.3.1.3'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.25.2.3.1.4'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.25.2.3.1.5'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.25.2.3.1.6'
inv_cisco_vlans: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.4.1.9.9.68.1.2.2.1.1'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.4.1.9.9.68.1.2.2.1.2'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.4.1.9.9.68.1.2.2.1.4'
inv_if: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.1'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.2'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.31.1.1.1.18'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.3'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.5'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.31.1.1.1.15'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.8'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.7'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.6'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.2.2.1.9'
snmp_extended_info: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.2'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.4'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.5'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.7'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.10'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.11'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.12'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.47.1.1.1.1.13'
snmp_os: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.1.1.0'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.1.2.0'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.1.3.0'
Running 'snmpbulkwalk -Cr15 -v2c -c *** -m "" -M "" -t 15.00 -r 4 -Cc -OQ -OU -On -Ot 10.10.15.244 .1.3.6.1.2.1.1.5.0'
Write data to cache file /omd/sites/***/tmp/check_mk/data_source_cache/snmp/checking/accsw04.***.is
Trying to acquire lock on /omd/sites/***/tmp/check_mk/data_source_cache/snmp/checking/accsw04.***.is
Got lock on /omd/sites/***/tmp/check_mk/data_source_cache/snmp/checking/accsw04.***.is
Releasing lock on /omd/sites/***/tmp/check_mk/data_source_cache/snmp/checking/accsw04.***.is
Released lock on /omd/sites/***/tmp/check_mk/data_source_cache/snmp/checking/accsw04.***.is
[cpu_tracking] Stop [7ff92ca4e430 - Snapshot(process=posix.times_result(user=0.10999999999999988, system=0.13, children_user=0.04, children_system=0.17, elapsed=0.9699999997392297))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7ff92ca4ea60]
[PiggybackFetcher] Fetch with cache settings: NoCache(accsw04.***.is, base_path=/omd/sites/***/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'accsw04.***.is'. Skip processing.
No piggyback files for '10.10.15.244'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7ff92ca4ea60 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.SNMP
Trying to acquire lock on /omd/sites/***/var/check_mk/persisted_sections/snmp/accsw04.***.is
Got lock on /omd/sites/***/var/check_mk/persisted_sections/snmp/accsw04.***.is
Releasing lock on /omd/sites/***/var/check_mk/persisted_sections/snmp/accsw04.***.is
Released lock on /omd/sites/***/var/check_mk/persisted_sections/snmp/accsw04.***.is
Stored persisted sections: cisco_cpu_multiitem, cisco_mem, cisco_power, cisco_secure, if64, snmp_info, snmp_uptime
Using persisted section SectionName('cisco_cpu_multiitem')
Using persisted section SectionName('cisco_mem')
Using persisted section SectionName('cisco_power')
Using persisted section SectionName('cisco_secure')
Using persisted section SectionName('if64')
Using persisted section SectionName('snmp_info')
Using persisted section SectionName('snmp_uptime')
  -> Add sections: ['cisco_cpu_multiitem', 'cisco_mem', 'cisco_power', 'cisco_secure', 'hr_mem', 'if64', 'inv_cisco_vlans', 'inv_if', 'snmp_extended_info', 'snmp_info', 'snmp_os', 'snmp_uptime']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7ff92c8d8400]
value store: synchronizing
Trying to acquire lock on /omd/sites/***/tmp/check_mk/counters/accsw04.***.is
Got lock on /omd/sites/***/tmp/check_mk/counters/accsw04.***.is
value store: loading from disk
Releasing lock on /omd/sites/***/tmp/check_mk/counters/accsw04.***.is
Released lock on /omd/sites/***/tmp/check_mk/counters/accsw04.***.is
CPU utilization 1    Utilization in the last 5 minutes: 7.00%
Interface 00001      [Vlan1], (up), MAC: 00:17:E0:BA:D3:C0, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10101      [GigabitEthernet0/1], (up), MAC: 00:17:E0:BA:D3:81, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10102      [GigabitEthernet0/2], (down), MAC: 00:17:E0:BA:D3:82, Speed: 1 GBit/s
Interface 10103      [GigabitEthernet0/3], (down), MAC: 00:17:E0:BA:D3:83, Speed: 1 GBit/s
Interface 10104      [GigabitEthernet0/4], (down), MAC: 00:17:E0:BA:D3:84, Speed: 1 GBit/s
Interface 10105      [GigabitEthernet0/5], (up), MAC: 00:17:E0:BA:D3:85, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10106      [GigabitEthernet0/6], (up), MAC: 00:17:E0:BA:D3:86, Speed: 100 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10107      [GigabitEthernet0/7], (up), MAC: 00:17:E0:BA:D3:87, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10108      [GigabitEthernet0/8], (down), MAC: 00:17:E0:BA:D3:88, Speed: 1 GBit/s
Interface 10109      [GigabitEthernet0/9], (up), MAC: 00:17:E0:BA:D3:89, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10110      [GigabitEthernet0/10], (down), MAC: 00:17:E0:BA:D3:8A, Speed: 1 GBit/s
Interface 10111      [GigabitEthernet0/11], (up), MAC: 00:17:E0:BA:D3:8B, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10112      [GigabitEthernet0/12], (up), MAC: 00:17:E0:BA:D3:8C, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10113      [GigabitEthernet0/13], (down), MAC: 00:17:E0:BA:D3:8D, Speed: 1 GBit/s
Interface 10114      [GigabitEthernet0/14], (up), MAC: 00:17:E0:BA:D3:8E, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10115      [GigabitEthernet0/15], (down), MAC: 00:17:E0:BA:D3:8F, Speed: 1 GBit/s
Interface 10116      [GigabitEthernet0/16], (up), MAC: 00:17:E0:BA:D3:90, Speed: 100 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10117      [GigabitEthernet0/17], (down), MAC: 00:17:E0:BA:D3:91, Speed: 1 GBit/s
Interface 10118      [GigabitEthernet0/18], (up), MAC: 00:17:E0:BA:D3:92, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10119      [GigabitEthernet0/19], (up), MAC: 00:17:E0:BA:D3:93, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10120      [GigabitEthernet0/20], (up), MAC: 00:17:E0:BA:D3:94, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10121      [GigabitEthernet0/21], (up), MAC: 00:17:E0:BA:D3:95, Speed: 100 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10122      [GigabitEthernet0/22], (down), MAC: 00:17:E0:BA:D3:96, Speed: 1 GBit/s
Interface 10123      [GigabitEthernet0/23], (down), MAC: 00:17:E0:BA:D3:97, Speed: 1 GBit/s
Interface 10124      [GigabitEthernet0/24], (down), MAC: 00:17:E0:BA:D3:98, Speed: 1 GBit/s
Interface 10125      [GigabitEthernet0/25], (down), MAC: 00:17:E0:BA:D3:99, Speed: 1 GBit/s
Interface 10126      [GigabitEthernet0/26], (down), MAC: 00:17:E0:BA:D3:9A, Speed: 1 GBit/s
Interface 10127      [GigabitEthernet0/27], (down), MAC: 00:17:E0:BA:D3:9B, Speed: 1 GBit/s
Interface 10128      [GigabitEthernet0/28], (down), MAC: 00:17:E0:BA:D3:9C, Speed: 1 GBit/s
Interface 10129      [GigabitEthernet0/29], (down), MAC: 00:17:E0:BA:D3:9D, Speed: 1 GBit/s
Interface 10130      [GigabitEthernet0/30], (down), MAC: 00:17:E0:BA:D3:9E, Speed: 1 GBit/s
Interface 10131      [GigabitEthernet0/31], (down), MAC: 00:17:E0:BA:D3:9F, Speed: 1 GBit/s
Interface 10132      [GigabitEthernet0/32], (down), MAC: 00:17:E0:BA:D3:A0, Speed: 1 GBit/s
Interface 10133      [GigabitEthernet0/33], (down), MAC: 00:17:E0:BA:D3:A1, Speed: 1 GBit/s
Interface 10134      [GigabitEthernet0/34], (down), MAC: 00:17:E0:BA:D3:A2, Speed: 1 GBit/s
Interface 10135      [GigabitEthernet0/35], (down), MAC: 00:17:E0:BA:D3:A3, Speed: 1 GBit/s
Interface 10136      [GigabitEthernet0/36], (down), MAC: 00:17:E0:BA:D3:A4, Speed: 1 GBit/s
Interface 10137      [GigabitEthernet0/37], (down), MAC: 00:17:E0:BA:D3:A5, Speed: 1 GBit/s
Interface 10138      [GigabitEthernet0/38], (down), MAC: 00:17:E0:BA:D3:A6, Speed: 1 GBit/s
Interface 10139      [GigabitEthernet0/39], (up), MAC: 00:17:E0:BA:D3:A7, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10140      [GigabitEthernet0/40], (down), MAC: 00:17:E0:BA:D3:A8, Speed: 1 GBit/s
Interface 10141      [GigabitEthernet0/41], (up), MAC: 00:17:E0:BA:D3:A9, Speed: 10 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10142      [GigabitEthernet0/42], (up), MAC: 00:17:E0:BA:D3:AA, Speed: 100 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10143      [GigabitEthernet0/43], (up), MAC: 00:17:E0:BA:D3:AB, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10144      [GigabitEthernet0/44], (up), MAC: 00:17:E0:BA:D3:AC, Speed: 100 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10145      [GigabitEthernet0/45], (up), MAC: 00:17:E0:BA:D3:AD, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10146      [GigabitEthernet0/46], (up), MAC: 00:17:E0:BA:D3:AE, Speed: 10 MBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10147      [GigabitEthernet0/47], (down), MAC: 00:17:E0:BA:D3:AF, Speed: 1 GBit/s
Interface 10148      [GigabitEthernet0/48], (down), MAC: 00:17:E0:BA:D3:B0, Speed: 1 GBit/s
Interface 10149      [GigabitEthernet0/49], (up), MAC: 00:17:E0:BA:D3:B1, Speed: 1 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 10150      [GigabitEthernet0/50], (down), MAC: 00:17:E0:BA:D3:B2, Speed: 1 GBit/s
Interface 10151      [GigabitEthernet0/51], (down), MAC: 00:17:E0:BA:D3:B3, Speed: 1 GBit/s
Interface 10152      [GigabitEthernet0/52], (down), MAC: 00:17:E0:BA:D3:B4, Speed: 1 GBit/s
Interface 14501      [Null0], (up), Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Mem used I/O         Usage: 77.73% - 9.32 MB of 11.99 MB
Mem used Processor   Usage: 26.18% - 24.16 MB of 92.30 MB
Port Security        No port security violation
Power Switch 1 PowerSupply 1 Status: normal, Source: AC
SNMP Info            Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2005 by Cisco Systems, Inc. Compiled Tue 30-Aug-05 14:19 by yenanh, accsw04.***
Uptime               Up since Jun 21 2023 10:36:00, Uptime: 1 day 5 hours
+ EXECUTING INVENTORY PLUGINS
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: skipped (no data)
 check_mk: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 inv_cisco_vlans: ok
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 inventory_checkmk: skipped (no data)
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_job_container: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 snmp_extended_info: ok
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_roles: skipped (no data)
 lnx_if: skipped (no data)
 lnx_if: skipped (no data)
 mem: skipped (no data)
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 snmp_info: ok
 snmp_info: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_container_node_name: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_os: ok
 snmp_os: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)

Trying to acquire lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is
Got lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is
Releasing lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is
Released lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is
Trying to acquire lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is.gz
Got lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is.gz
Releasing lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is.gz
Released lock on /omd/sites/***/tmp/check_mk/status_data/accsw04.***.is.gz
Trying to acquire lock on /omd/sites/***/tmp/check_mk/status_data/.last
Got lock on /omd/sites/***/tmp/check_mk/status_data/.last
Releasing lock on /omd/sites/***/tmp/check_mk/status_data/.last
Released lock on /omd/sites/***/tmp/check_mk/status_data/.last
No piggyback files for 'accsw04.***.is'. Skip processing.
No piggyback files for '10.10.15.244'. Skip processing.
[cpu_tracking] Stop [7ff92c8d8400 - Snapshot(process=posix.times_result(user=0.09999999999999964, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.10999999940395355))]
[snmp] Success, execution time 1.1 sec | execution_time=1.080 user_time=0.210 system_time=0.140 children_user_time=0.040 children_system_time=0.170 cmk_time_snmp=0.520 cmk_time_agent=0.000

I’m having issues with multiple switches in my environment and they regularly report absurd usage on the switches interfaces, 10’s of GBit/s on a 1GBit port and even up to TBit/s on some of them. Anyone have any clue what could be causing this?

Switch
Switch782×226 10.7 KB

Edit: I know Rocky 8 isn’t officially supported but I switched last week from Centos7 and it was the same problem there.

2

I thought it was just me. I never had the problem on 2.1.0p13 but when I upgraded to p25 it started. I have a dashboard that shows WAN Utilization on a bunch of firewalls. When it does happen, it’ll happen to several at exactly the same time in the early afternoon CST (-6). By your pic, it looks like they are happening at very close time to mine.
I have since upgraded to 2.2 and no difference.
I figured it was getting confused during some routine maintenance.

In the image below, the top firewall has a maximum throughput of 1Gbps. In the bottom, it consistently passes under 1Mbps (load balanced unit as this connection is not used except basically for heartbeat.

Always at the same time of day, but random on which device shows it.
Screenshot 2023-06-22 134949

3

Just happened again on 2 of 15 firewalls…

Top and bottom in the image. Top one reporting just under 700Mbps, and 123Mbps down on the bottom. Those numbers are not possible.

Screenshot 2023-06-22 144658
Screenshot 2023-06-22 144658118×819 12.3 KB

4

So we might have found a solution or atleast why this is happening. The spikes seems to be at the same time as HW/SW inventory check.

Can you see if it happens during that time for you?

5

Yes, the time lines up for me as well, almost to the minute. Good find!

6

Checkmk does not have a lot of logic for when there are delay for SNMP and that the data is complete.

7

@Birkir,

can you set the Normal check interval for service checks for the Check_MK HW/SW Inventory for this device to e.g. 10 minutes and confirm that you see spikes every 10 minutes please ?

I am working on a similar issue for a customer:
He also get s impossible Interface spikes but in this case the device was not reachable for about 15min for the monitoring system. I was not able to reproduce this behaviour so far.

Can you also provide one rrd file of an interface showing impossible spikes ?
Because what I observed in the mentioned case is that not only one data point was impossibly high but 3 in a row.

8

JFYI, there is another topic that is perhaps related

https://forum.checkmk.com/t/used-bandwidth-on-specific-interface-averaged-over-5-mins/40115

I suspect that checkmk sometimes miscalculates the values but I can’t proof it so far nor reproduce.

9

@Birkir , @genitguy3

Are you using Average values for used bandwidth of the rule Network interfaces and switch ports on the interfaces where you see the spikes ?

10

I am not using the Average values option in that rule.

11

I tried that and everything was normal.
But I found something else. It doesn’t seem to be the HW/SW inventory alone. I turned on the periodic service discovery and set it to 10 minutes and I get the spikes again, and if I turn off the HW/SW inventory the spikes continue.
Is anyone else able to confirm that its the Periodic service discovery that is causing this?
Here are the rulechanges I made

rulechanges
rulechanges1306×639 53 KB

12

I can only post 1 picture at a time.
And the Spikes that happen when the rules are activated/deactivated.

spikes_0
spikes_0703×207 15 KB

13

But looks also a little bit like too few datapoints (only one every 5min ?)

I have a report from another user, that this is not related to HW/SW inventory and he is also not using Average values for used bandwidth.

Give me some time to verify.
I assume you check intervall for the Interfaces is 1min ?

14

I am still unable to reproduce such spikes in my lab.
Even not with turning on and off the periodic service discovery.

But I now extended the checkmk interface check lib/python3/cmk/base/plugins/agent_based/utils/interfaces.py to log what it got as raw data and what it calculated out of that.

Please send me a private message if you are interested to uses this to debug your interface spike phenomenas.

Some background:

The bandwidth rate is calculated using the previous value, and it’s timestamp from the value store.

rates
rates802×522 209 KB

The modified interfaces.py will create a folder structure interface_logs/yyy/mm/dd//<interface_fs_save>.log and log a json line one each check intervall

OMD[central]:~$ tail -n1 interface_logs/2023/08/16/myhost.mycompany.tld/tun0.log | jq ''
{
  "timestamp": 1692195397.2286897,
  "host_name": "myhost.mycompany.tld",
  "item": "tun0",
  "in_octets": 136673920,    <--- raw rx counter data
  "out_octets": 24733351,   <--- raw tx counter data
  "value_store": {
    "outtraffic.None": [
      1692195397.2286897,
      24733351
    ],  
    "intraffic.None": [
      1692195397.2286897,
      136673920
    ]   
  },  
  "rates_dict": {
    "intraffic": 2037.9487204080037,
    "outtraffic": 750.7709995843549
  }
}
  • Storing this data helps to do a back in time analysis.
  • Logs are stored in ~/interface_log/yyyy/mm/dd folders to allow you to easily remove old data from by a cronjob:
echo "0 6 * * * find ~/interface_logs/ -type f -mtime +4 -delete" > ~/etc/cron.d/custom_interface_logs
omd restart crontab
15

It only looks like few datapoints, it was polling every minute, and i put the service check every 10 minutes, I don’t remember why I have the spike at 10:37.

But I will check out the script and put it into my environment and see what the logs say.

16

I did not have to wait long for some data, what is the best way to share the data?

17

If you managed to install my modified interfaces.py and it logged to
~/interface_logs/<YYYY>/<MM>/<DD>/<host>/<interface>.log while you observed a spike, please send me a private message here in the forum and I will get in contact with you.
I need the logfile and also the rrd file of the interface on which you see the spike.

18

Okay, I now analysed one set of logs files collected by the modified interfaces.py and it indicates
that the device/switch reported a rx/tx value that was about 1500 times higher as the previous and following record.

And checkmk just did it’s job and calculated a rate out of it.
Looks like the the problem is on the switch not in checkmk for this case at least.

20

I may have found a solution, and it’s not connected to any rule.
In my case atleast it was the virtual machine hosting CheckMK was underpowered, but it was not visible in any logs. We use VMware in my environment and it showed that the machine was barely going over 50%, but I have been emailing with Jodok and he mentioned something about memory and it might be capping that for just 1-2 seconds, but it wasn’t that but I wondered if the CPU was enough so I brought up htop and saw that when the spikes happened the cpu was at 100% on all available cores which was 6, so I upped the amount to 8 and the spikes stopped, and the CPU is not at 100% on all cores anymore.
And that is just the problem with average values over any amount of time, I have been dealing with a similar situation in my environment. In that case it was a switch dropping packets, but nothing showed on the logs except that packets were being dropped. Turns out that the switch port was going over 1gbit and was dropping the frames, I could only see it with Wireshark and going down to bits per millisecond.

tl;dr: Check if the spikes stop if you give the machine more CPU power.

21

Hi @jodok.glabasna, are you still willing to look at some captured logs? I don’t seem to have the option to message you directly, would you be able to flick me pm i can reply to?