Dell IDRAC 7 not getting all info

brononius · February 18, 2020, 9:29am

Hey,

In the past, my dell IDRAC 7 server (audrey) was getting all necessary info about RAID config, memory moduels, cpu…
But lately I reinstalled my check_mk server, and since then, I’m not able anymore to get this info.
My bookmarks are still in place (backup/restore). That’s why I’m sure that I did it direclty with the IDRAC IP.

For the setting in WATO, I’m having:

Check_MK Agent: No agent
SNMP: SNMP v2 or V3
SNMP Credentials: XXX

With this, i’m getting some info, but not all:

cmk -D audrey

Addresses: 192.168.222.244
Tags:
Host groups:
Contact groups:
Type of agent: TCP (port: 6556)
Is aggregated: no
Services:
checktype item params description groups summarized to groups

SNMP is working:

snmpwalk -v 2c -c COMMUNITY audrey

iso.3.6.1.2.1.1.1.0 = “”
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.674.10892.5
iso.3.6.1.2.1.1.3.0 = Timeticks: (316338) 0:52:43.38
iso.3.6.1.2.1.1.4.0 = STRING: “"support@dell.com"”
iso.3.6.1.2.1.1.5.0 = STRING: “audrey”
iso.3.6.1.2.1.1.6.0 = STRING: “"unknown"”
iso.3.6.1.2.1.1.8.0 = Timeticks: (4) 0:00:00.04
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.2.1.49
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.4
iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.2.1.50
…
iso.3.6.1.2.1.55.1.5.1.2.1 = STRING: “lo”
iso.3.6.1.2.1.55.1.5.1.2.11 = STRING: “bond0.777”
iso.3.6.1.2.1.55.1.5.1.3.1 = OID: ccitt.0
iso.3.6.1.2.1.55.1.5.1.3.11 = OID: ccitt.0
iso.3.6.1.2.1.55.1.5.1.4.1 = Gauge32: 16436
iso.3.6.1.2.1.55.1.5.1.4.11 = Gauge32: 1500
iso.3.6.1.2.1.55.1.5.1.8.1 = “”
iso.3.6.1.2.1.55.1.5.1.8.11 = Hex-STRING: 5C F9 DD F3 DE 4F
iso.3.6.1.2.1.55.1.5.1.9.1 = INTEGER: 1
iso.3.6.1.2.1.55.1.5.1.9.11 = INTEGER: 1

Any idea what I’m missing?

Thanks a lot !!!

tavanez · February 18, 2020, 9:38am

Hi @brononius,

for instance didn’t you miss some plugin ?

Oh I pasted the url just as example.

Cheers

brononius · February 18, 2020, 9:49am

I downloaded the mib and had moved this to /opt/omd/versions/default/share/snmp/mibs, but guess I’m still missing some parts.
Fe I don’t have the file check_idrac_2.py somewhere. And I don’t see this somewhere on the github.

tavanez · February 18, 2020, 9:55am

The mibs are used to event console translation if I’m not mistaken, you should had a specific plugin for it.

Did you backed up your old cmk server ? If so try to find there if you had the plugin.

brononius · February 18, 2020, 9:57am

Sadly not. Just the config from WATO.

tavanez · February 18, 2020, 9:58am

check this out ! I didn’t test this one.

oh and yes one of the requirements for this check to work is to save the mibs

brononius · February 18, 2020, 11:10am

Getting close, but not yet there…

What I’ve done so far:

cd /root
git clone GitHub - dangmocrang/check_idrac: A script to monitoring DELL IDRAC via SNMP
cd check_idrac
cp idrac_2.2rc4 /usr/lib/check_mk_agent/plugins/idrac_2.2rc4.py
cp idrac-smiv2.mib /opt/omd/sites/MYSITE/local/share/check_mk/mibs/.

When I run now the command, I’m getting an issue with the mibs.

/idrac_2.2rc4.py -H audrey -v2c -c COMMUNITY | more

your MIB may out of dated!
error - MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/
ietf:/usr/share/mibs/netsnmp. Cannot find module (SNMPv2-TC): At line 7 in /usr/share/snmp/mibs/UCD-IPFWACC-MIB.txt. Cannot find module (SNMPv2-SMI): At line 34 in /usr/share/snmp/mibs/UCD-SNMP-
MIB.txt. Cannot find module (SNMPv2-TC): At line 37 in /usr/share/snmp/mibs/UCD-SNMP-MIB.txt. Did not find ‘enterprises’ in module #-1 (/usr/share/snmp/mibs/UCD-SNMP-MIB.txt). Did not find ‘Disp
layString’ in module #-1 (/usr/share/snmp/mibs/UCD-SNMP-MIB.txt). Did not find ‘TruthValue’ in module #-1 (/usr/share/snmp/mibs/UCD-SNMP-MIB.txt). Unlinked OID in UCD-SNMP-MIB: ucdavis ::= { ent
erprises 2021 }. Undefined identifier: enterprises near line 39 of /usr/share/snmp/mibs/UCD-SNMP-MIB.txt. Did not find ‘DisplayString’ in module #-1 (/usr/share/snmp/mibs/UCD-IPFWACC-MIB.txt). D
id not find ‘ucdExperimental’ in module UCD-SNMP-MIB (/usr/share/snmp/mibs/UCD-IPFWACC-MIB.txt). Unlinked OID in UCD-IPFWACC-MIB: ucdIpFwAccMIB ::= { ucdExperimental 1 }. Undefined identifier: u
cdExperimental near line 11 of /usr/share/snmp/mibs/UCD-IPFWACC-MIB.txt. Did not find ‘ucdExperimental’ in module UCD-SNMP-MIB (/usr/share/snmp/mibs/UCD-IPFILTER-MIB.txt). Unlinked OID in UCD-IP
FILTER-MIB: ucdIpFilter ::= { ucdExperimental 2 }. Undefined identifier: ucdExperimental near line 9 of /usr/share/snmp/mibs/UCD-IPFILTER-MIB.txt. Did not find ‘ucdavis’ in module UCD-SNMP-MIB (
/usr/share/snmp/mibs/UCD-DEMO-MIB.txt). Unlinked OID in UCD-DEMO-MIB: ucdDemoMIB ::= { ucdavis 14 }. Undefined identifier: ucdavis near line 7 of /usr/share/snmp/mibs/UCD-DEMO-MIB.txt. Cannot fi
nd module (SNMPv2-SMI): At line 4 in /usr/share/snmp/mibs/IANA-RTPROTO-MIB.txt. Did not find ‘mib-2’ in module #-1 (/usr/share/snmp/mibs/IANA-RTPROTO-MIB.txt). Unlinked OID in IANA-RTPROTO-MIB:
ianaRtProtoMIB ::= { mib-2 84 }. Undefined identifier: mib-2 near line 7 of /usr/share/snmp/mibs/IANA-RTPROTO-MIB.txt. Cannot find module (SNMPv2-SMI): At line 4 in /usr/share/snmp/mibs/IANAifTy
pe-MIB.txt. Did not find ‘mib-2’ in module #-1 (/usr/share/snmp/mibs/IANAifType-MIB.txt). Unlinked OID in IANAifType-MIB: ianaifType ::= { mib-2 30 }. Undefined identifier: mib-2 near line 7 of
/usr/share/snmp/mibs/IANAifType-MIB.txt. Cannot find module (SNMPv2-TC): At line 10 in /usr/share/snmp/mibs/UCD-DLMOD-MIB.txt. Did not find ‘DisplayString’ in module #-1 (/usr/share/snmp/mibs/UC
D-DLMOD-MIB.txt). Did not find ‘ucdExperimental’ in module UCD-SNMP-MIB (/usr/share/snmp/mibs/UCD-DLMOD-MIB.txt). Unlinked OID in UCD-DLMOD-MIB: ucdDlmodMIB ::= { ucdExperimental 14 }. Undefined
identifier: ucdExperimental near line 13 of /usr/share/snmp/mibs/UCD-DLMOD-MIB.txt. Cannot find module (SNMPv2-SMI): At line 5 in /usr/share/snmp/mibs/IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt. Did n
ot find ‘mib-2’ in module #-1 (/usr/share/snmp/mibs/IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt). Unlinked OID in IANA-ADDRESS-FAMILY-NUMBERS-MIB: ianaAddressFamilyNumbers ::= { mib-2 72 }. Undefined id
entifier: mib-2 near line 8 of /usr/share/snmp/mibs/IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt. Cannot find module (SNMPv2-SMI): At line 8 in /usr/share/snmp/mibs/NET-SNMP-MIB.txt. Did not find ‘enterp
rises’ in module #-1 (/usr/share/snmp/mibs/NET-SNMP-MIB.txt). Unlinked

tavanez · February 18, 2020, 11:12am

You didn’t read the manual
Just kidding, see if it helps, according to your output it seems so.

Requires:

This check included with DELL IDRAC MIB file (required). You can copy MIB file to your default MIB folder, usually is /usr/share/snmp/mibs/. If you don’t want to do this, keep it where you want and use option -m/–mib with absolute path to it.

brononius · February 18, 2020, 11:22am

grrr, indeed. I copied it into 5 different mib-folders, except the /usr/share/snmp/mibs/.
Now it works much better.

./idrac_2.2rc4.py -H audrey -v2c -c COMMUNITIY

PS
– PS 1: OK, Volt I/O: 264 V/230 V, Current: 0.8 A, Watt I/O: 594.0 W/495 W
– PS 2: OK, Volt I/O: 264 V/234 V, Current: 0.4 A, Watt I/O: 594.0 W/495 W

…

Next step is now telling checkmk to use this plugin?
I just removed the host, and made a new one, but it’s not picking up the plugin/mib…

tavanez · February 18, 2020, 11:26am

Setup a classic active and passive Monitoring checks, there you’ll specify the host and the command line (which will be the command you posted).
Then try a cmk -nvII hostname (the -n don’t submit results to core) and see if the checks are ok

then let me know if it working.

brononius · February 18, 2020, 12:00pm

Took me a bit, till I understood that I needed to give in the full path.

/usr/lib/check_mk_agent/plugins/check_idrac_2.py -H audrey -v2c -c l3z3r

Sadly, in cli, I’m not a ‘positive’ result:

cmk -nvII audrey
Hostname or tag specification ‘audrey’ does not match any host.

In the GUI, It looks promising:

check_idrac_2.py
PS --PS 1: OK, Volt I/O: 264 V/230 V, Current: 0.8 A, Watt I/O: 594.0 W/495 W --PS 2: OK, Volt I/O: 264 V/234 V, Current: 0.4 A, Watt I/O: 594.0 W/495 W DISK --PDisk 1 (0:1:0) 1862.5 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [ATA, HDD, S/N: ] isFailing: 0 --PDisk 2 (0:1:4) 1862.5 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [ATA, HDD, S/N: ] isFailing: 0 --PDisk 3 (0:1:5) 4657.0 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [ATA, HDD, S/N: ] isFailing: 0 --PDisk 4 (0:1:1) 4657.0 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [ATA, HDD, S/N: ] isFailing: 0 --PDisk 5 (0:1:2) 1117.25 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [SEAGATE, HDD, S/N: Z4021DXL] isFailing: 0 --PDisk 6 (0:1:6) 1117.25 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [SEAGATE, HDD, S/N: Z402239R] isFailing: 0 --PDisk 7 (0:1:3) 1117.25 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [HP, HDD, S/N: ] isFailing: 0 --PDisk 8 (0:1:7) 1117.25 GB: ONLINE, PowerStat: SPUNUP, HotSpare: no [HP, HDD, S/N: ] isFailing: 0 FAN --System Board Fan1: 3840 RPM - ENABLED/OK --System Board Fan2: 3840 RPM - ENABLED/OK --System Board Fan3: 3840 RPM - ENABLED/OK --System Board Fan4: 3840 RPM - ENABLED/OK --System Board Fan5: 3960 RPM - ENABLED/OK --System Board Fan6: 3960 RPM - ENABLED/OK BATTERY --System Board CMOS Battery: ENABLED/OK [PRESENCEDETECTED] --PERC1 ROMB Battery: ENABLED/OK [PRESENCEDETECTED] --PERC2 ROMB Battery: ENABLED/OK [0] PU --PU 1: ENABLED/OK, RedundancyStatus: FULL, SystemBoard Pwr Consumption: 252 W MEM --Memory 1 (DIMM Socket A1) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE27D] --Memory 2 (DIMM Socket A2) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 0F46DE1A] --Memory 3 (DIMM Socket B1) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Samsung, S/N: 129A1B04] --Memory 4 (DIMM Socket B2) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE21E] --Memory 5 (DIMM Socket A3) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 35D148C8] --Memory 6 (DIMM Socket A4) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 35D148AA] --Memory 7 (DIMM Socket A5) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 421CE54A] --Memory 8 (DIMM Socket A6) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 85BDCEDF] --Memory 9 (DIMM Socket B3) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE30D] --Memory 10 (DIMM Socket B4) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2CD] --Memory 11 (DIMM Socket B5) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE1B1] --Memory 12 (DIMM Socket B6) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2CA] --Memory 13 (DIMM Socket A7) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 35D14802] --Memory 14 (DIMM Socket A8) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 86DB40AE] --Memory 15 (DIMM Socket A9) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE29C] --Memory 16 (DIMM Socket A10) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE21C] --Memory 17 (DIMM Socket A11) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 35D147D4] --Memory 18 (DIMM Socket A12) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 35D148A7] --Memory 19 (DIMM Socket B7) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2AD] --Memory 20 (DIMM Socket B8) 16.0 GB/1600 MHz: ENABLED/OK [DDR3, Samsung, S/N: 85BDCEE4] --Memory 21 (DIMM Socket B9) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2C8] --Memory 22 (DIMM Socket B10) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2AC] --Memory 23 (DIMM Socket B11) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 391CE2D0] --Memory 24 (DIMM Socket B12) 16.0 GB/1333 MHz: ENABLED/OK [DDR3, Micron Technology, S/N: 4525723F] VDISK --VDisk 1 (VD1): OK/ONLINE, RAID-1 (1862.5 GB), BadBlock: 0 [Virtual Disk 0 on Integrated RAID Controller 1] --VDisk 2 (Data): OK/ONLINE, RAID-1 (4657.0 GB), BadBlock: 0 [Virtual Disk 1 on Integrated RAID Controller 1] --VDisk 3 (VD3): OK/ONLINE, RAID-1 (1117.25 GB), BadBlock: 0 [Virtual Disk 2 on Integrated RAID Controller 1] --VDisk 4 (VD4): OK/ONLINE, RAID-1 (1117.25 GB), BadBlock: 0 [Virtual Disk 3 on Integrated RAID Controller 1] GLOBAL --systemStateGlobalSystemStatus OK SENSOR --System Board Inlet Temp: 13.0 C ENABLED/OK --System Board Exhaust Temp: 30.0 C ENABLED/OK --CPU1 Temp: 46.0 C ENABLED/OK --CPU2 Temp: 54.0 C ENABLED/OK CPU --CPU 1 (6 cores/12 threads): ENABLED/OK [Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz] --CPU 2 (6 cores/12 threads): ENABLED/OK [Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz]

tavanez · February 18, 2020, 12:02pm

In cli use the ip maybe, strange…

But if you try to discover the services do they appear ?

brononius · February 18, 2020, 12:11pm

With IP, I’m getting the same result:

cmk -nvII 192.168.222.244
Hostname or tag specification ‘192.168.222.244’ does not match any host.

ping audrey
PING audrey (192.168.222.244) 56(84) bytes of data.
64 bytes from audrey (192.168.222.244): icmp_seq=1 ttl=64 time=0.411 ms

For the services, I’m getting the feeling that he sees it as 1?

tavanez · February 18, 2020, 12:27pm

It does indeed I would look into the plugin or the GitHub page in order to see if its possible to call with different arguments in order to create “n” services.
Right now I can’t but I’ll take a look later

@brononius take a look after “Group scan” : check_idrac/MANUAL.md at master · dangmocrang/check_idrac · GitHub

It happens that you can configure some arguments to fetch specific metrics, not sure if it will fully adapt to want you want but at least it will give you more visibility.
You’ll just need to create more than one rule for each metric.
Or if it works as the http checks you can use some voodoo regex in just one rule to do them all

brononius · February 18, 2020, 12:54pm

Thanks!

Will have a look how to create checks for each item…

ps with my previous installation, I didn’t created this. It was done automatically in check_mk.
When I check the plugins, I see that some Dell servers are in there (fe OpenManger, poweredge, idrac…). Guess that somewhere, I need to tell check_mk that he needs to link this with this server?

tavanez · February 18, 2020, 2:08pm

Well it depends on what default checks the cmk have and the ones developed by you (colleague/company/etc).
If you hadn’t it configured this way, maybe you had some custom checks developed that were in use.
EDIT: Yes I’ve seen what you are saying. It should be detected automatically by check_mk once you do the full scan. It shouldn’t be needed to create any rule for that. Strange

Have the IDRAC been updated meanwhile for instance?

brononius · February 18, 2020, 4:58pm

No. It runs fine, so I keep it like that.

I just reinstalled the monitoring server with check_mk.
This version has been changed (OS and check_mk).

Is there any way I can ‘force’ the check plugin with the host?

tavanez · February 18, 2020, 5:05pm

As far as I’m concerned no, it should discover them and insert internal cmk magic aka know the oid and show the services for the existing checks. So I would wait for anymore opinions cause I can’t help you more.

@ricardoftribeiro can you help ?

ricardoftribeiro · February 18, 2020, 5:29pm

@tavanez i can’t even understand what the issue is, and what’s being asked.

OS & CMK were reinstalled, bunch of another steps were earlier made, i just lost myself in the process.

@brononius - can you make a macro resume about what you did and in what point you’re now ?

Cheers,

brononius · February 18, 2020, 5:47pm

Not sure anymore why I reinstalled the server. At some point, he crashed, and I went for a complete reinstall. This was done some months ago. And lately, I noticed that the checks for my RAID configs isn’t done anymore.

In short:

Backup taken over WATO
Installation of OS (Ubuntu 19.10)
Installation of Check_MK (currently 1.6.0p2)
Restore of backup
My server (PowerEdge R720 with Dell Idrac 7) is nicely reachable over snmp.
Nothing was changed on this server, beside some changes for testing (fe v2 to v3 and back).
With tavanez, I’ve add a manual python scripts and mibs. And this all seemed to worked (in cli).
All info is getting in when I run ‘/usr/lib/check_mk_agent/plugins/check_idrac_2.py -H audrey -v2c -c l3z3r’
But when I try to add a ‘manual check’ in checkmk, it’s not been split between all checks.
I then recalled that I didn’t do this with my old installation (the manual checks).
And after checking the ‘check plugins’, several SNMP checks exist for Dell (openmanage, idrac, poweredge…).
So somewhere, the link isn’t made between the plugin and my host.

Thanks already for your patience and help !!!