CMK 2.3p39, redfish 2.3.77 , Dell iDRAC fw 7.20.60.50 services keep vanishing /reappearing

Troubleshooting
1

CMK version: 2.3p39
OS version: virt1

there seems to be an odd behaviour with Dell iDRAC fw 7.20.60.50, as certain services keep vanishing (→ UNKN) and after some check attempts they are available again undtil a few checks later … and so on

Most affected services are Temp, Fan, Memory and Harddrives

Has anyone seen this too?

BR
Thomas

2

The problem here is i think the iDRAC version. iDRAC 7 only supports some Redfish functions and not all. With iDRAC 7 it is the same problem as with iLO4 some are working without any problem and some shown such things like your UNKN state.

If i have such a problematic interface inside my monitoring - i only select the system state to be fetched. If there is a problem the rollup state changes and i need to have a look at the management interface directly.

I think there is no real solution for problems with older management interfaces.

3

Hi andi

do you mean iDRAC version or iDRAC fw version?

Best

Thomas

4

Sorry - your system is a iDRAC 9 or?

If it is iDRAC 9 and you have from time to time the UNKN for some services and later than again the OK pls check the following things.

Firmware Inventory needs to be done with a high cache time or if not needed disable it.

image
image378×177 6.91 KB

And for the CheckMK service of this devices it can be good to set the maximum timeout to 120 seconds. I had some Dell systems with quick response if you only fetch the normal data and with Firmware inventory it took over 1 minute.

5

Hi andi

yeah iDRAC9 with fw 7.xxx.xxx but it seems not all of those are affected.
Atm it seems that it makes a difference how many hops the iDRACs are away from the Checkmk.

We wil try out and report back ^^

Best
Tom

6

Hi,

running 2.5.0p8 pro edition on rhel9 here and have similar things happening.

I know it is now default baked in into checkmk, but i’d give it a go…

so redfish → idrac 7.30.10.50 (enterprise license), services keep on vanishing (mainly fan’s)

if i go to my url directly:

/redfish/v1/Chassis/System.Embedded.1£
–>{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/EnvironmentMetrics”},“EnvironmentalClass”:“A2”,“Id”:“System.Embedded.1”,“IndicatorLED”:“Lit”,“IndicatorLED@Redfish.Deprecated”:“Please migrate to use LocationIndicatorActive property”,“Links”:{“ComputerSystems”:[{“@odata.id”:“/redfish/v1/Systems/System.Embedded.1”}],“ComputerSystems@odata.count”:1,“Contains”:[],“Contains@odata.count”:0,“CooledBy”:[{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/0”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/1”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/2”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/3”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/4”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/5”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/6”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/7”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/8”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/9”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/10”},{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal#/Fans/11”}],“CooledBy@odata.count”:12,“Drives”:[{“@odata.id”:“/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.0-0:BOSS.SL.12-1”},{“@odata.id”:“/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.1-1:BOSS.SL.12-1”}],“Drives@odata.count”:2,“ManagedBy”:

- →
“Power@Redfish.Deprecated”:“Please migrate to use /redfish/v1/Chassis/System.Embedded.1/PowerSubsystem”,“PowerState”:“On”,“PowerSubsystem”:{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/PowerSubsystem”},“PoweredByParent”:false,“SKU”:“xxxxxx”,“Sensors”:{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Sensors”},“SerialNumber”:“xxxxxxx”,“Status”:{“Health”:“OK”,“HealthRollup”:“OK”,“State”:“Enabled”},“Thermal”:{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/Thermal”},“Thermal@Redfish.Deprecated”:“Please migrate to use /redfish/v1/Chassis/System.Embedded.1/ThermalSubsystem”,“ThermalManagedByParent”:false,“ThermalSubsystem”:{“@odata.id”:“/redfish/v1/Chassis/System.Embedded.1/ThermalSubsystem”},“TrustedComponents”:

when i run (somehwere between a huge chunk of data, it displays:
cmk -d hostname:
<<<redfish_thermal:sep(0)>>>
{“error”: “Thermal data could not be fetched\n”}
<<<redfish_power:sep(0)>>>

so, dell changed something in its new firmware, but i think checkmk still needs to follow? bug-worthy?

ofcourse, googled and AI’ed a bit:
The schema is Chassis.v1_25_1, which means a recent iDRAC firmware.

  • Thermal is marked deprecated: “Please migrate to use /redfish/v1/Chassis/System.Embedded.1/ThermalSubsystem”. Power is deprecated in favor of PowerSubsystem too.
  • Links.CooledBy still lists 12 fans, as Thermal#/Fans/0 to #/Fans/11. The chassis itself still knows about the fans.

The Checkmk Redfish special agent reads fans from the legacy Thermal endpoint. Newer iDRAC firmware may have emptied or trimmed Fans there, or dropped fields such as Name or Reading. The discovery plugin would then find nothing. Newer firmware moves the data to ThermalSubsystem/ThermalMetrics and ThermalSubsystem/Fans/<id>.

thanks for the feedback!

7

Same information as all the time before. Please use the Redfish-Mockup-Creator for a dump of your interface.
With this dump it is possible to check what information is available on your system and what’s not. You can sent this dump to me (not an Checkmk employee) directly or make a support ticket as you have the enterprise version.

8

Hi Andreas, will do that. early next week (I hope).

thanks for the feedback!