“Item not found in monitoring data” Custom Plugin Stopped working

Hello There,

I made a custom plugin to monitor fortigate BGP Peers it was working good in 2.0 until recently we had a peer ip change. But after that CheckMK 2.0 is not able to discover that service properly.

(we have both versions of checkmk 1.6 and 2.0 as we are still in transition to new version.) I noticed older version discovered it properly. But its not working for new version.

I checked under service configuration Once I disable the UNKN service in the disabled state it is showing proper data. but I move it to monitored servers it is showing “Item not found in monitoring data”.

The same plugin was working few days back now it stopped working not sure why. We havent make any changes from CheckMK side. Any suggestions to fix this issue?

Thanks,
Shivdev

Hi shivdev14,

did you perchance make an update of your Fortigate? I can remember Fortigate changing some of the OIDs in an update and having to adjust the checks accordingly. (It was regarding the av/ips signatures)

What does an snmpbulkwalk reveal?

It would be good to know what plugin is used (github/exchange source) and a little bit of the error output.
The output you can get with an “cmk --debug -vvn hostname” on the command line.
You can also do an “cmk --debug -vvII hostname” to see if at discovery time you have error messages.

@andreas-doehler @simon-mueller Thank you for reply. Sorry I know this got delayed. Had to prioritize other tasks. Again the issue is still the same. But this time i will provide more info.

Here is the plugin

from .agent_based_api.v1 import *
import pprint

bgp_peer_inventory_adminstates = [ 2 ]

def discover_bgp(section):
    for identifier, peerstate, adminstate, version, localaddr, \
        remoteaddr, remoteas, peerinupdates, peeroutupdates, lasterror, peeresttime in section:
        yield Service(item=remoteaddr)

def sec2hr(seconds):
    m, s = divmod(seconds, 60)
    h, m = divmod(m, 60)
    d, h = divmod(h, 24)
    hr = "%dd %02dh %02dm %02ds" % (d, h, m, s)
    return(hr)


def bgp_peerstate(st):
    names = {   '0' : 'none',
                '1' : 'idle',
                '2' : 'connect',
                '3' : 'active',
                '4' : 'opensnet',
                '5' : 'openconfirm',
                '6' : 'established' }
    return names.get(st, st)

def bgp_adminstate(st):
    names = {   '1' : 'stop',
                '2' : 'start', }
    return names.get(st, st)

def bgp_errors(key1, key2):
    names = {}
    names["0"] = { "0" : "NO ERROR"}
    names["1"] = { "0" : "Message",
                   "2" : "Connection Not Synchronized",
                   "3" : "Bad Message Length",
                   "4" : "Bad Message Type",
                  }
    names["2"] = { "0" : "OPEN",
                   "1" : "Unsupported Version Number",
                   "2" : "Bad Peer AS",
                   "3" : "Bad BGP Identifier",
                   "4" : "Unsupported Optional Parameter",
                   "5" : "Authentication Failure",
                   "6" : "Unacceptable Hold",
                  }
    names["3"] = { "0" : "UPDATE",
                   "1" : "Malformed Attribute List",
                   "2" : "Unrecognized Well-known Attribute",
                   "3" : "Missing Well-known Attribute",
                   "4" : "Attribute Flags Error",
                   "5" : "Attribute Length Error",
                   "6" : "Invalid ORIGIN Attribute",
                   "7" : "AS Routing Loop",
                   "8" : "Invalid NEXT_HOP Attribute",
                   "9" : "Optional Attribute Error",
                   "10" : "Invalid Network Field",
                   "11" : "Malformed AS_PATH",
                 }
    names["4"] = { "0" : "Hold Timer Expired",}

    names["5"] = { "0" : "Finite State Machine Error",}

    names["6"] = { "0" : "Administratively Shutdown",
                   "1" : "Max Prefix Reached",
                   "2" : "Peer Unconfigured",
                   "3" : "Administratively Reset",
                   "4" : "Connection Rejected",
                   "5" : "Other Configuration Change",
                  }
    return names[key1].get(key2)


def ByteToHex(byteStr):
    return ''.join( [ "%02X " % ord(x) for x in byteStr ] ).strip()

def check_bgp(item, section):
    #print(len(section))
    length_range = len(section)
    for i in range(length_range):
        # print(i)
        # print(section[i])
        for identifier, peerstate, adminstate, version, localaddr, \
            remoteaddr, remoteas, peerinupdates, peeroutupdates, lasterror, peeresttime in section:
            # print(localaddr, remoteaddr, item)
            if remoteaddr == item:
               output = "BGP Peer: %s Adminstate: %s " % (identifier, bgp_adminstate(str(adminstate)))
               lasterrorhex = ByteToHex(lasterror)
               byte1, byte2 = lasterrorhex.split()
               lerr = int(byte1, 16)
               lsuberr = int(byte2, 16)
               lasterrorstr = bgp_errors(str(lerr), str(lsuberr))
               peeresttimehr = sec2hr(int(peeresttime))
               #print(typeof(peerstate))
               if int(peerstate) == 1:
                   severity = "CRITICAL"
                   s = State.CRIT
                   output += "PeerState: %s(!!) Version: %s LocalAddr: %s \
                           RemoteAddr: %s RemoteAS: %s LastError: %s PeerEstablishedTime: %s" % \
                           (bgp_peerstate(str(peerstate)), version, localaddr, remoteaddr, remoteas, lasterrorstr, peeresttimehr )
               elif int(peerstate) in ['2', '3', '4', '5']:
                   severity = "WARNING"
                   s = State.WARN
                   output += "PeerState: %s(!) Version: %s LocalAddr: %s \
                           RemoteAddr: %s RemoteAS: %s LastError: %s PeerEstablishedTime: %s" % \
                           (bgp_peerstate(str(peerstate)), version, localaddr, remoteaddr, remoteas, lasterrorstr, peeresttimehr )
               elif int(peerstate) == 6:
                   severity = "OK"
                   s = State.OK
                   output += "PeerState: %s Version: %s LocalAddr: %s \
                           RemoteAddr: %s RemoteAS: %s LastError: %s PeerEstablishedTime: %s" % \
                           (bgp_peerstate(str(peerstate)), version, localaddr, remoteaddr, remoteas, lasterrorstr, peeresttimehr )
               else:
                   severity = "UNKNOWN"
                   s = State.UNKNOWN
                   output = "Invalid Output from Agent"

               infotext = severity + " - " + output

               section = section.pop(i)

               yield from check_levels(int(peeresttime), metric_name = 'peer_est_time')
               yield Metric(name="Peerinupdates", value=int(peerinupdates))
               yield Metric(name="Peeroutupdates", value=int(peeroutupdates))
               yield Metric(name="Peerstate", value=int(peerstate))

               yield Result(
                   state = s,
                   summary = infotext)

               return

            # return(3, "UNKNOWN - item not found in snmp data")


register.snmp_section(
    name = "fortigate_bgp_peer",
    detect = exists(".1.3.6.1.2.1.1.1.0"),
    fetch = SNMPTree(
        base = '.1.3.6.1.2.1.15.3.1',
        oids = [
            '1',  # "bgpPeerIdentifier"
            '2',  # "bgpPeerState"
            '3',  # "bgpPeerAdminStatus"
            '4',  # "bgpPeerNegotiatedVersion"
            '5',  # "bgpPeerLocalAddr"
            '7',  # "bgpPeerRemoteAddr"
            '9',  # "bgpPeerRemoteAs"
            '10', # "bgpPeerInUpdates"
            '11', # "bgpPeerOutUpdates"
            '14', # "bgpPeerLastError"
            '16', # "bgpPeerFsmEstablishedTime"
        ],
    ),
)

register.check_plugin(
    name = "fortigate_bgp_peer",
    service_name = "BGP Peer %s",
    discovery_function = discover_bgp,
    check_function = check_bgp
)

Here is the output I get from CLI

OMD[~~~]:~$ cmk --debug --detect-plugins=fortigate_bgp_peer -v ~~~~~~~
Checkmk version 2.0.0p3
+ FETCHING DATA
[SNMPFetcher] Execute data source
No piggyback files for '~~~~~~~'. Skip processing.
No piggyback files for '10.0.129.10'. Skip processing.
[PiggybackFetcher] Execute data source
+ PARSE FETCHER RESULTS
Received no piggyback data
BGP Peer 10.214.112.1 750919.00, OK - BGP Peer: 155.130.6.229 Adminstate: start PeerState: established Version: 4 LocalAddr: 10.214.112.10                            RemoteAddr: 10.214.112.1 RemoteAS: 65001 LastError: Hold Timer Expired PeerEstablishedTime: 8d 16h 35m 19s
BGP Peer 10.214.120.1 214301.00, OK - BGP Peer: 155.130.6.229 Adminstate: start PeerState: established Version: 4 LocalAddr: 10.214.120.10                            RemoteAddr: 10.214.120.1 RemoteAS: 65001 LastError: Hold Timer Expired PeerEstablishedTime: 2d 11h 31m 41s
BGP Peer 10.215.112.1 1850272.00, OK - BGP Peer: 209.170.215.19 Adminstate: start PeerState: established Version: 4 LocalAddr: 10.215.112.10                            RemoteAddr: 10.215.112.1 RemoteAS: 65001 LastError: NO ERROR PeerEstablishedTime: 21d 09h 57m 52s
BGP Peer 10.215.120.1 214300.00, OK - BGP Peer: 209.170.215.19 Adminstate: start PeerState: established Version: 4 LocalAddr: 10.215.120.10                            RemoteAddr: 10.215.120.1 RemoteAS: 65001 LastError: Hold Timer Expired PeerEstablishedTime: 2d 11h 31m 40s
[snmp] Success, execution time 0.6 sec | execution_time=0.570 user_time=0.250 system_time=0.050 children_user_time=0.000 children_system_time=0.000 cmk_time_snmp=0.270 cmk_time_agent=0.000

In the UI I can see these services under service configuration like below

But in the Host status detail I can’t see this

I think the issue is inside the plugin can you please point out where I did a mistake?

Thank you,
Shivdevkumar

Will it be anything related to CheckMK Version? In our Test CheckMK which is in Version 2.0.0p8 (CEE) where it works good. But in our PROD version 2.0.0p3 it throws this error – Item not found in monitoring data.

Still can you please verify the plugin if I did any mistakes there?

Thank you,
Shivdev

this is where I think i made a mistake. Is there any other you think we can do this. In section we get list of list where there will 7-8 BGP Peers with for each peer there is a list of 11 items. So how should we parse through list and go to the next one??