Redfish Extension isn't working since updating to CMK 2.3.0p23

redl_jan · January 9, 2025, 10:52am

Since we updated Checkmk to 2.3.0p23 and the Redfish extension to 2.3.69 afterwards, none of our iLO checks are working anymore and all of them has changed to the status UNKNOWN.

CMK version:
2.3.0p23

OS version:
Checkmk virt1 Mark I - Version: 1.7.8

Server Hardware
HPE Synergy 480 Gen10 Plus

iLO Version
iLO 5
3.04 Apr 17 2024

Redfish Extension Version
2.3.69

Error message:

CRIT	[special_redfish]: ERROR: too many retries for connection attempt:CRIT
OK	[piggyback]: Success (but no data found for this host)

Connection-Test – Output for Agent

Agent exited with code 1: INFO 2025-01-09 11:38:40 redfish: Redfish API
INFO 2025-01-09 11:38:40 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 11:38:41 redfish.rest.v1: Response Time for GET to /redfish/v1: 0.8140705469995737 seconds.
INFO 2025-01-09 11:38:41 redfish.rest.v1: Attempt 1 of /redfish/v1/SessionService/Sessions
INFO 2025-01-09 11:38:44 redfish.rest.v1: Retrying /redfish/v1/SessionService/Sessions [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 11:38:45 redfish.rest.v1: Attempt 2 of /redfish/v1/SessionService/Sessions
INFO 2025-01-09 11:38:46 redfish.rest.v1: Response Time for GET to /redfish/v1/SessionService/Sessions: 1.0170456339983502 seconds.
INFO 2025-01-09 11:38:46 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 11:38:46 redfish.rest.v1: Response Time for GET to /redfish/v1: 0.02799262299959082 seconds.
INFO 2025-01-09 11:38:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Managers?$expand=.
INFO 2025-01-09 11:38:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Managers?$expand=.: 0.07114569899567869 seconds.
INFO 2025-01-09 11:38:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems
INFO 2025-01-09 11:38:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems: 0.012778231990523636 seconds.
INFO 2025-01-09 11:38:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1
INFO 2025-01-09 11:38:49 redfish.rest.v1: Retrying /redfish/v1/Systems/1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 11:38:50 redfish.rest.v1: Attempt 2 of /redfish/v1/Systems/1
INFO 2025-01-09 11:38:51 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1: 1.3251638140063733 seconds.
Traceback (most recent call last):
File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish", line 13, in <module>
sys.exit(main())
^^^^^^
File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 873, in main
return special_agent_main(parse_arguments, agent_redfish_main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 173, in special_agent_main
return _special_agent_main_core(parse_arguments, main_fn, argv or sys.argv[1:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 149, in _special_agent_main_core
return main_fn(args)
^^^^^^^^^^^^^
File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 863, in agent_redfish_main
get_information(redfishobj)
File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 619, in get_information
firmwares = fetch_data(
^^^^^^^^^^^
File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 186, in fetch_data
response_url = redfishobj.redfish_connection.get(url, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: RestClientBase.get() got an unexpected keyword argument 'timeout'

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

value store: synchronizing
Trying to acquire lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Got lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
value store: loading from disk
Releasing lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Released lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Checkmk version 2.3.0p23
+ FETCHING DATA
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='special_redfish', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fa76b472360]
Read from cache: AgentFileCache(<<HOSTNAME>>, path_template=/omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/{hostname}, max_age=MaxAge(checking=0, discovery=45.0, inventory=45.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 7376 sec, allowed is 0 sec)
Calling: /omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish -u <<API-USER>> --password-id <<PASSWORD-ID>>:/omd/sites/<<SITE-NAME>>/var/check_mk/passwords_merged -p 443 -P https --debug <<IP-ADDRESS>>
Get data from program
[cpu_tracking] Stop [7fa76b472360 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.29, children_system=0.02, elapsed=0.8099999986588955))]
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fa7692fc770]
Read from cache: NoCache(<<HOSTNAME>>, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
Get piggybacked data
[cpu_tracking] Stop [7fa7692fc770 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
[cpu_tracking] Start [7fa76ac3d460]
+ PARSE FETCHER RESULTS
  HostKey(hostname='<<HOSTNAME>>', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
CPU 1                PEND Check plug-in received no monitoring data
CPU 2                PEND Check plug-in received no monitoring data
Drive 0-960GB 6G SATA SSD PEND Check plug-in received no monitoring data
Drive 1-480GB 16G NVMe SSD PEND Check plug-in received no monitoring data
Drive 1I:1:1         PEND Check plug-in received no monitoring data
Drive 1I:1:2         PEND Check plug-in received no monitoring data
Drive 2-480GB 16G NVMe SSD PEND Check plug-in received no monitoring data
Drive 2-960GB 6G SATA SSD PEND Check plug-in received no monitoring data
Fan 1                PEND Check plug-in received no monitoring data
Memory Summary       PEND Check plug-in received no monitoring data
Memory proc1dimm1    PEND Check plug-in received no monitoring data
Memory proc1dimm10   PEND Check plug-in received no monitoring data
Memory proc1dimm12   PEND Check plug-in received no monitoring data
Memory proc1dimm14   PEND Check plug-in received no monitoring data
Memory proc1dimm16   PEND Check plug-in received no monitoring data
Memory proc1dimm3    PEND Check plug-in received no monitoring data
Memory proc1dimm5    PEND Check plug-in received no monitoring data
Memory proc1dimm7    PEND Check plug-in received no monitoring data
Memory proc2dimm1    PEND Check plug-in received no monitoring data
Memory proc2dimm10   PEND Check plug-in received no monitoring data
Memory proc2dimm12   PEND Check plug-in received no monitoring data
Memory proc2dimm14   PEND Check plug-in received no monitoring data
Memory proc2dimm16   PEND Check plug-in received no monitoring data
Memory proc2dimm3    PEND Check plug-in received no monitoring data
Memory proc2dimm5    PEND Check plug-in received no monitoring data
Memory proc2dimm7    PEND Check plug-in received no monitoring data
Network adapter 1    PEND Check plug-in received no monitoring data
Network adapter DC082000 PEND Check plug-in received no monitoring data
Physical port 141    PEND Check plug-in received no monitoring data
Physical port 142    PEND Check plug-in received no monitoring data
Physical port 145    PEND Check plug-in received no monitoring data
Physical port 146    PEND Check plug-in received no monitoring data
Physical port 147    PEND Check plug-in received no monitoring data
Physical port 148    PEND Check plug-in received no monitoring data
Storage Controller 18 PEND Check plug-in received no monitoring data
Storage controller DE00A000 PEND Check plug-in received no monitoring data
Storage controller DE07A000 PEND Check plug-in received no monitoring data
System state         PEND Check plug-in received no monitoring data
Temperature 01-Inlet Ambient PEND Check plug-in received no monitoring data
Temperature 02-CPU 1 PEND Check plug-in received no monitoring data
Temperature 03-CPU 2 PEND Check plug-in received no monitoring data
Temperature 04-P1 DIMM 1-8 PEND Check plug-in received no monitoring data
Temperature 06-P1 DIMM 9-16 PEND Check plug-in received no monitoring data
Temperature 08-P2 DIMM 1-8 PEND Check plug-in received no monitoring data
Temperature 10-P2 DIMM 9-16 PEND Check plug-in received no monitoring data
Temperature 12-VR P1 PEND Check plug-in received no monitoring data
Temperature 13-VR P2 PEND Check plug-in received no monitoring data
Temperature 14-VR P1 Mem 1 PEND Check plug-in received no monitoring data
Temperature 15-VR P1 Mem 2 PEND Check plug-in received no monitoring data
Temperature 16-VR P2 Mem 1 PEND Check plug-in received no monitoring data
Temperature 17-VR P2 Mem 2 PEND Check plug-in received no monitoring data
Temperature 18-Chipset PEND Check plug-in received no monitoring data
Temperature 19-BMC   PEND Check plug-in received no monitoring data
Temperature 20-HD Max PEND Check plug-in received no monitoring data
Temperature 22-Stor Batt PEND Check plug-in received no monitoring data
Temperature 23-E-Fuse PEND Check plug-in received no monitoring data
Temperature 24.1-HD Controller-Add-in card PEND Check plug-in received no monitoring data
Temperature 24.2-HD Controller-I/O controlle PEND Check plug-in received no monitoring data
Temperature 24.3-HD Controller-I/O controlle PEND Check plug-in received no monitoring data
Temperature 24.4-HD Controller-Add-in card PEND Check plug-in received no monitoring data
Temperature 25.1-HD Cntlr Zone-Add-in card PEND Check plug-in received no monitoring data
Temperature 25.2-HD Cntlr Zone-Add-in card PEND Check plug-in received no monitoring data
Temperature 26-I/O Zone 1 PEND Check plug-in received no monitoring data
Temperature 27-I/O Zone 2 PEND Check plug-in received no monitoring data
Temperature 28-I/O Zone 3 PEND Check plug-in received no monitoring data
Temperature 29-P1 Mem Zone 1 PEND Check plug-in received no monitoring data
Temperature 30-P1 Mem Zone 2 PEND Check plug-in received no monitoring data
Temperature 35.1-Mezz 3-I/O module PEND Check plug-in received no monitoring data
Temperature 35.2-Mezz 3-External Environment PEND Check plug-in received no monitoring data
Temperature 36.1-Mezz 3 Inlet-External Envir PEND Check plug-in received no monitoring data
Temperature 37-Sys Exhaust 1 PEND Check plug-in received no monitoring data
Temperature 38-Sys Exhaust 2 PEND Check plug-in received no monitoring data
Temperature 39.1-ExpBayBoot-I/O controller PEND Check plug-in received no monitoring data
Temperature 39.2-ExpBayBoot-I/O controller PEND Check plug-in received no monitoring data
Volume 1             PEND Check plug-in received no monitoring data
Volume 65            PEND Check plug-in received no monitoring data
Volume 73            PEND Check plug-in received no monitoring data
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
[cpu_tracking] Stop [7fa76ac3d460 - Snapshot(process=posix.times_result(user=4.83, system=0.03, children_user=0.0, children_system=0.0, elapsed=4.8599999994039536))]
[special_redfish] TypeError: RestClientBase.get() got an unexpected keyword argument 'timeout'(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 5.7 sec | execution_time=5.680 user_time=4.830 system_time=0.030 children_user_time=0.290 children_system_time=0.020 cmk_time_ds=0.500 cmk_time_agent=0.010
Agent exited with code 1: INFO 2025-01-09 11:30:04 redfish: Redfish API
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1: 0.3137007810000796 seconds.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1/SessionService/Sessions
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1/SessionService/Sessions: 0.008461257006274536 seconds.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1: 0.027933414006838575 seconds.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1/Managers?$expand=.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1/Managers?$expand=.: 0.10660394800652284 seconds.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems: 0.008216526999603957 seconds.
INFO 2025-01-09 11:30:04 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1
INFO 2025-01-09 11:30:04 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1: 0.05993620400840882 seconds.
Traceback (most recent call last):
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish", line 13, in <module>
    sys.exit(main())
             ^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 873, in main
    return special_agent_main(parse_arguments, agent_redfish_main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 173, in special_agent_main
    return _special_agent_main_core(parse_arguments, main_fn, argv or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 149, in _special_agent_main_core
    return main_fn(args)
           ^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 863, in agent_redfish_main
    get_information(redfishobj)
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 619, in get_information
    firmwares = fetch_data(
                ^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 186, in fetch_data
    response_url = redfishobj.redfish_connection.get(url, timeout=timeout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: RestClientBase.get() got an unexpected keyword argument 'timeout'(!!)

andreas-doehler · January 9, 2025, 11:12am

As this interface looks really slow, please deactivate the firmware section inside the special agent config or you can try to adjust the timeout value.
How long took the check_mk run before the update?

redl_jan · January 9, 2025, 12:19pm

Thank you so much!
Yes, this did the trick for all iLOs, except one.

You are right, the execution time is insane.
Here is the debug log of the host where the problem persists.
The execution time was 44.6 sec

What’s the expected amount of time this run should take?

value store: synchronizing
Trying to acquire lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Got lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
value store: loading from disk
Releasing lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Released lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Checkmk version 2.3.0p23
+ FETCHING DATA
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='special_redfish', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f98df984440]
Read from cache: AgentFileCache(<<HOSTNAME>>, path_template=/omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/{hostname}, max_age=MaxAge(checking=0, discovery=45.0, inventory=45.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 13789 sec, allowed is 0 sec)
Calling: /omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish -u <<API-USER>> --password-id <<PASSWORD-ID>>:/omd/sites/<<SITE-NAME>>/var/check_mk/passwords_merged -p 443 -P https -n FirmwareInventory --debug <<IP-ADDRESS>>
Get data from program
[cpu_tracking] Stop [7f98df984440 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.47, children_system=0.03, elapsed=39.439999997615814))]
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f98dda06b10]
Read from cache: NoCache(<<HOSTNAME>>, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
Get piggybacked data
[cpu_tracking] Stop [7f98dda06b10 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f98df309070]
+ PARSE FETCHER RESULTS
  HostKey(hostname='<<HOSTNAME>>', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
CPU 1                PEND Check plug-in received no monitoring data
CPU 2                PEND Check plug-in received no monitoring data
Drive 0-960GB 6G SATA SSD PEND Check plug-in received no monitoring data
Drive 1-480GB 16G NVMe SSD PEND Check plug-in received no monitoring data
Drive 1I:1:1         PEND Check plug-in received no monitoring data
Drive 1I:1:2         PEND Check plug-in received no monitoring data
Drive 2-480GB 16G NVMe SSD PEND Check plug-in received no monitoring data
Drive 2-960GB 6G SATA SSD PEND Check plug-in received no monitoring data
Fan 1                PEND Check plug-in received no monitoring data
Memory Summary       PEND Check plug-in received no monitoring data
Memory proc1dimm1    PEND Check plug-in received no monitoring data
Memory proc1dimm10   PEND Check plug-in received no monitoring data
Memory proc1dimm12   PEND Check plug-in received no monitoring data
Memory proc1dimm14   PEND Check plug-in received no monitoring data
Memory proc1dimm16   PEND Check plug-in received no monitoring data
Memory proc1dimm3    PEND Check plug-in received no monitoring data
Memory proc1dimm5    PEND Check plug-in received no monitoring data
Memory proc1dimm7    PEND Check plug-in received no monitoring data
Memory proc2dimm1    PEND Check plug-in received no monitoring data
Memory proc2dimm10   PEND Check plug-in received no monitoring data
Memory proc2dimm12   PEND Check plug-in received no monitoring data
Memory proc2dimm14   PEND Check plug-in received no monitoring data
Memory proc2dimm16   PEND Check plug-in received no monitoring data
Memory proc2dimm3    PEND Check plug-in received no monitoring data
Memory proc2dimm5    PEND Check plug-in received no monitoring data
Memory proc2dimm7    PEND Check plug-in received no monitoring data
Network adapter 1    PEND Check plug-in received no monitoring data
Network adapter DC082000 PEND Check plug-in received no monitoring data
Physical port 147    PEND Check plug-in received no monitoring data
Physical port 148    PEND Check plug-in received no monitoring data
Physical port 151    PEND Check plug-in received no monitoring data
Physical port 152    PEND Check plug-in received no monitoring data
Physical port 153    PEND Check plug-in received no monitoring data
Physical port 154    PEND Check plug-in received no monitoring data
Storage Controller 18 PEND Check plug-in received no monitoring data
Storage controller DE00A000 PEND Check plug-in received no monitoring data
Storage controller DE07A000 PEND Check plug-in received no monitoring data
System state         PEND Check plug-in received no monitoring data
Temperature 01-Inlet Ambient PEND Check plug-in received no monitoring data
Temperature 02-CPU 1 PEND Check plug-in received no monitoring data
Temperature 03-CPU 2 PEND Check plug-in received no monitoring data
Temperature 04-P1 DIMM 1-8 PEND Check plug-in received no monitoring data
Temperature 06-P1 DIMM 9-16 PEND Check plug-in received no monitoring data
Temperature 08-P2 DIMM 1-8 PEND Check plug-in received no monitoring data
Temperature 10-P2 DIMM 9-16 PEND Check plug-in received no monitoring data
Temperature 12-VR P1 PEND Check plug-in received no monitoring data
Temperature 13-VR P2 PEND Check plug-in received no monitoring data
Temperature 14-VR P1 Mem 1 PEND Check plug-in received no monitoring data
Temperature 15-VR P1 Mem 2 PEND Check plug-in received no monitoring data
Temperature 16-VR P2 Mem 1 PEND Check plug-in received no monitoring data
Temperature 17-VR P2 Mem 2 PEND Check plug-in received no monitoring data
Temperature 18-Chipset PEND Check plug-in received no monitoring data
Temperature 19-BMC   PEND Check plug-in received no monitoring data
Temperature 20-HD Max PEND Check plug-in received no monitoring data
Temperature 22-Stor Batt PEND Check plug-in received no monitoring data
Temperature 23-E-Fuse PEND Check plug-in received no monitoring data
Temperature 24.1-HD Controller-Add-in card PEND Check plug-in received no monitoring data
Temperature 24.2-HD Controller-I/O controlle PEND Check plug-in received no monitoring data
Temperature 24.3-HD Controller-I/O controlle PEND Check plug-in received no monitoring data
Temperature 24.4-HD Controller-Add-in card PEND Check plug-in received no monitoring data
Temperature 25.1-HD Cntlr Zone-Add-in card PEND Check plug-in received no monitoring data
Temperature 25.2-HD Cntlr Zone-Add-in card PEND Check plug-in received no monitoring data
Temperature 26-I/O Zone 1 PEND Check plug-in received no monitoring data
Temperature 27-I/O Zone 2 PEND Check plug-in received no monitoring data
Temperature 28-I/O Zone 3 PEND Check plug-in received no monitoring data
Temperature 29-P1 Mem Zone 1 PEND Check plug-in received no monitoring data
Temperature 30-P1 Mem Zone 2 PEND Check plug-in received no monitoring data
Temperature 35.1-Mezz 3-I/O module PEND Check plug-in received no monitoring data
Temperature 35.2-Mezz 3-External Environment PEND Check plug-in received no monitoring data
Temperature 36.1-Mezz 3 Inlet-External Envir PEND Check plug-in received no monitoring data
Temperature 37-Sys Exhaust 1 PEND Check plug-in received no monitoring data
Temperature 38-Sys Exhaust 2 PEND Check plug-in received no monitoring data
Temperature 39.1-ExpBayBoot-I/O controller PEND Check plug-in received no monitoring data
Temperature 39.2-ExpBayBoot-I/O controller PEND Check plug-in received no monitoring data
Volume 1             PEND Check plug-in received no monitoring data
Volume 65            PEND Check plug-in received no monitoring data
Volume 73            PEND Check plug-in received no monitoring data
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
[cpu_tracking] Stop [7f98df309070 - Snapshot(process=posix.times_result(user=5.1899999999999995, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=5.1900000013411045))]
[special_redfish] redfish.rest.v1.RetriesExhaustedError(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 44.6 sec | execution_time=44.630 user_time=5.190 system_time=0.020 children_user_time=0.470 children_system_time=0.030 cmk_time_ds=38.930 cmk_time_agent=0.000
Agent exited with code 1: INFO 2025-01-09 13:04:19 redfish: Redfish API
INFO 2025-01-09 13:04:19 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 13:04:22 redfish.rest.v1: Retrying /redfish/v1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:23 redfish.rest.v1: Attempt 2 of /redfish/v1
INFO 2025-01-09 13:04:25 redfish.rest.v1: Response Time for GET to /redfish/v1: 2.0466752469947096 seconds.
INFO 2025-01-09 13:04:25 redfish.rest.v1: Attempt 1 of /redfish/v1/SessionService/Sessions
INFO 2025-01-09 13:04:25 redfish.rest.v1: Response Time for GET to /redfish/v1/SessionService/Sessions: 0.028292276998399757 seconds.
INFO 2025-01-09 13:04:25 redfish.rest.v1: Attempt 1 of /redfish/v1
INFO 2025-01-09 13:04:25 redfish.rest.v1: Response Time for GET to /redfish/v1: 0.02743115500197746 seconds.
INFO 2025-01-09 13:04:25 redfish.rest.v1: Attempt 1 of /redfish/v1/Managers?$expand=.
INFO 2025-01-09 13:04:28 redfish.rest.v1: Retrying /redfish/v1/Managers?$expand=. [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:29 redfish.rest.v1: Attempt 2 of /redfish/v1/Managers?$expand=.
INFO 2025-01-09 13:04:31 redfish.rest.v1: Response Time for GET to /redfish/v1/Managers?$expand=.: 1.1080565099982778 seconds.
INFO 2025-01-09 13:04:31 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems
INFO 2025-01-09 13:04:31 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems: 0.01303032599389553 seconds.
INFO 2025-01-09 13:04:31 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1
INFO 2025-01-09 13:04:34 redfish.rest.v1: Retrying /redfish/v1/Systems/1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:35 redfish.rest.v1: Attempt 2 of /redfish/v1/Systems/1
INFO 2025-01-09 13:04:36 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1: 1.1066690479929093 seconds.
INFO 2025-01-09 13:04:36 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/Storage
INFO 2025-01-09 13:04:36 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/Storage: 0.011801622997154482 seconds.
INFO 2025-01-09 13:04:36 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/Storage/DE00A000
INFO 2025-01-09 13:04:39 redfish.rest.v1: Retrying /redfish/v1/Systems/1/Storage/DE00A000 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:40 redfish.rest.v1: Attempt 2 of /redfish/v1/Systems/1/Storage/DE00A000
INFO 2025-01-09 13:04:41 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/Storage/DE00A000: 1.0155670320091303 seconds.
INFO 2025-01-09 13:04:41 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/Storage/DE07A000
INFO 2025-01-09 13:04:41 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/Storage/DE07A000: 0.01486969500547275 seconds.
INFO 2025-01-09 13:04:41 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces
INFO 2025-01-09 13:04:44 redfish.rest.v1: Retrying /redfish/v1/Systems/1/EthernetInterfaces [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:45 redfish.rest.v1: Attempt 2 of /redfish/v1/Systems/1/EthernetInterfaces
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces: 1.0146630230010487 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/141
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/141: 0.00793724499817472 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/142
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/142: 0.013767169002676383 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/143
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/143: 0.01347801300289575 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/144
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/144: 0.012261159994523041 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/145
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/145: 0.013098452996928245 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/EthernetInterfaces/146
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/EthernetInterfaces/146: 0.012505519000114873 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/NetworkInterfaces
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/NetworkInterfaces: 0.012160291997133754 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/NetworkInterfaces/DC082000
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/NetworkInterfaces/DC082000: 0.011982734009507112 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/Memory
INFO 2025-01-09 13:04:46 redfish.rest.v1: Response Time for GET to /redfish/v1/Systems/1/Memory: 0.01478053699247539 seconds.
INFO 2025-01-09 13:04:46 redfish.rest.v1: Attempt 1 of /redfish/v1/Systems/1/Memory/proc1dimm1
INFO 2025-01-09 13:04:49 redfish.rest.v1: Retrying /redfish/v1/Systems/1/Memory/proc1dimm1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:50 redfish.rest.v1: Attempt 2 of /redfish/v1/Systems/1/Memory/proc1dimm1
INFO 2025-01-09 13:04:53 redfish.rest.v1: Retrying /redfish/v1/Systems/1/Memory/proc1dimm1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
INFO 2025-01-09 13:04:54 redfish.rest.v1: Attempt 3 of /redfish/v1/Systems/1/Memory/proc1dimm1
INFO 2025-01-09 13:04:57 redfish.rest.v1: Retrying /redfish/v1/Systems/1/Memory/proc1dimm1 [HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)]
Traceback (most recent call last):
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/ssl.py", line 1252, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3.12/ssl.py", line 1104, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/redfish/rest/v1.py", line 863, in _rest_request
    resp = self._session.request(method.upper(), "{}{}".format(self.__base_url, reqpath), data=body,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/requests/adapters.py", line 578, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='<<IP-ADDRESS>>', port=443): Read timed out. (read timeout=3)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish", line 13, in <module>
    sys.exit(main())
             ^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 873, in main
    return special_agent_main(parse_arguments, agent_redfish_main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 173, in special_agent_main
    return _special_agent_main_core(parse_arguments, main_fn, argv or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 149, in _special_agent_main_core
    return main_fn(args)
           ^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 863, in agent_redfish_main
    get_information(redfishobj)
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 645, in get_information
    fetch_sections(redfishobj, resulting_sections, redfishobj.sections, system)
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 271, in fetch_sections
    result = fetch_collection(redfishobj, section_data, section)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 207, in fetch_collection
    element_data = fetch_data(redfishobj, element.get("@odata.id"), component)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/special_agents/agent_redfish.py", line 188, in fetch_data
    response_url = redfishobj.redfish_connection.get(url, None)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/redfish/rest/v1.py", line 620, in get
    return self._rest_request(path, method='GET', args=args,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/redfish/rest/v1.py", line 1055, in _rest_request
    return super(HttpClient, self)._rest_request(path=path, method=method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/<<SITE-NAME>>/local/lib/python3/redfish/rest/v1.py", line 907, in _rest_request
    raise RetriesExhaustedError() from cause_exception
redfish.rest.v1.RetriesExhaustedError(!!)

andreas-doehler · January 10, 2025, 10:57am

This system has so many timeouts. There is something completely wrong with this iLO.
What iLO version is this?

redl_jan · January 10, 2025, 11:59am

Same Version as the other ones.

iLO 5, Version: 3.04 Apr 17 2024
License Type Advanced

andreas-doehler · January 10, 2025, 1:35pm

Is it the same behavior after you rebootet the iLO?

redl_jan · January 14, 2025, 11:01am

After an iLO reboot, the execution time is 38.6 sec now.
So, it got slightly faster, but the cmk run still times out at the exact same step.

Since there are no more issues with the other iLOs now, there seems to be a problem with this specific iLO indeed.
I’m gonna try the error-causing API request manually with Postman, to confirm that.

andreas-doehler · January 14, 2025, 3:54pm

I don’t think that we have here one specific API command with an problem.
In your log file there are already timeouts at the very first fetch request → /redfish/v1
The answer for this should be instant but your log shows one timeout and the second try needed 2 seconds.
That is way way too long.

redl_jan · February 3, 2025, 6:16am

This weekend, we updated our HPE Synergy Systems including their iLOs.
Now we run iLO 5, 3.09 Oct 08 2024 on our Blades.

It seems that this has solved the problem.
The run is successful now.

Thank you so much for your help!

Best regards
Jan

value store: synchronizing
Trying to acquire lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Got lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
value store: loading from disk
Releasing lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Released lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/counters/<<HOSTNAME>>
Checkmk version 2.3.0p23
+ FETCHING DATA
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='special_redfish', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f3b9b0657c0]
Read from cache: AgentFileCache(<<HOSTNAME>>, path_template=/omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/{hostname}, max_age=MaxAge(checking=0, discovery=45.0, inventory=45.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 15 sec, allowed is 0 sec)
Calling: /omd/sites/<<SITE-NAME>>/local/lib/python3/cmk_addons/plugins/redfish/libexec/agent_redfish -u <<API-USER>> --password-id <<PASSWORD-ID>>:/omd/sites/<<SITE-NAME>>/var/check_mk/passwords_merged -p 443 -P https -n FirmwareInventory --debug <<IP-ADDRESS>>
Get data from program
Write data to cache file /omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/<<HOSTNAME>>
Trying to acquire lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/<<HOSTNAME>>
Got lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/<<HOSTNAME>>
Releasing lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/<<HOSTNAME>>
Released lock on /omd/sites/<<SITE-NAME>>/tmp/check_mk/data_source_cache/special_redfish/<<HOSTNAME>>
[cpu_tracking] Stop [7f3b9b0657c0 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.36, children_system=0.04, elapsed=3.230000000447035))]
  Source: SourceInfo(hostname='<<HOSTNAME>>', ipaddress='<<IP-ADDRESS>>', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f3b9b7838f0]
Read from cache: NoCache(<<HOSTNAME>>, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
Get piggybacked data
[cpu_tracking] Stop [7f3b9b7838f0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f3b9b0e83b0]
+ PARSE FETCHER RESULTS
<<<check_mk:sep(32)>>> / Transition NOOPParser -> HostSectionParser
<<<redfish_manager:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_system:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_chassis:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_storage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_ethernetinterfaces:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_memory:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_networkinterfaces:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_processors:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_drives:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_volumes:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_hostbusadapters:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_arraycontrollers:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_physicaldrives:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_networkadapters:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_power:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<redfish_thermal:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='<<HOSTNAME>>', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'redfish_arraycontrollers', 'redfish_chassis', 'redfish_drives', 'redfish_ethernetinterfaces', 'redfish_hostbusadapters', 'redfish_manager', 'redfish_memory', 'redfish_networkadapters', 'redfish_networkinterfaces', 'redfish_physicaldrives', 'redfish_power', 'redfish_processors', 'redfish_storage', 'redfish_system', 'redfish_thermal', 'redfish_volumes']
  HostKey(hostname='<<HOSTNAME>>', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
CPU 1                Type: CPU, Model: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz, Cores: 28, Threads: 56, Speed maximum 4000 MHz
CPU 2                Type: CPU, Model: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz, Cores: 28, Threads: 56, Speed maximum 4000 MHz
Drive 0-960GB 6G SATA SSD Size: 894GB, Speed 6.0 Gbs, Media Life Left: 100%
Drive 1-480GB 16G NVMe SSD Size: 447GB, Speed 16.0 Gbs, Media Life Left: 100%
Drive 1I:1:1         Size: 894GB, Speed 6.0 Gbs
Drive 1I:1:2         Size: 894GB, Speed 6.0 Gbs
Drive 2-480GB 16G NVMe SSD Size: 447GB, Speed 16.0 Gbs, Media Life Left: 100%
Drive 2-960GB 6G SATA SSD Size: 894GB, Speed 6.0 Gbs, Media Life Left: 100%
Fan 1                Speed: 18.0%
Memory Summary       Capacity: 1024GB, with State: Rollup State: Normal
Memory proc1dimm1    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm10   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm12   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm14   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm16   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm3    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm5    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc1dimm7    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm1    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm10   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm12   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm14   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm16   Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm3    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm5    Size: 64GB, Type: DDR4-3200 MultiBitECC
Memory proc2dimm7    Size: 64GB, Type: DDR4-3200 MultiBitECC
Network adapter 1    Model: Synergy 6820C 25/50Gb CNA, SeNr: <<SERIAL-NUMBER>>, PartNr: P02424-001
Network adapter DC082000 Model: Synergy 6820C 25/50Gb CNA, SeNr: <<SERIAL-NUMBER>>, PartNr: P02054-B21
Physical port 147    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:02
Physical port 148    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:03
Physical port 151    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:06
Physical port 152    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:07
Physical port 153    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:08
Physical port 154    Link: LinkUp, Speed: 50000Mbps, MAC: 32:4a:1d:c0:00:09
Storage Controller 18 Type: HPE Smart Array P204i-c SR Gen10, Serial: <<SERIAL-NUMBER>> 
Storage controller DE00A000 Everything looks OK - 1 detail available
Storage controller DE07A000 Everything looks OK - 1 detail available
System state         System with SerialNr: <<SERIAL-NUMBER>>, has State: Component State: Normal, Rollup State: Normal, This resource is enabled.
Temperature 01-Inlet Ambient Temperature: 17.0 °C
Temperature 02-CPU 1 Temperature: 40.0 °C
Temperature 03-CPU 2 Temperature: 40.0 °C
Temperature 04-P1 DIMM 1-8 Temperature: 31.0 °C
Temperature 06-P1 DIMM 9-16 Temperature: 31.0 °C
Temperature 08-P2 DIMM 1-8 Temperature: 36.0 °C
Temperature 10-P2 DIMM 9-16 Temperature: 36.0 °C
Temperature 12-VR P1 Temperature: 42.0 °C
Temperature 13-VR P2 Temperature: 54.0 °C
Temperature 14-VR P1 Mem 1 Temperature: 26.0 °C
Temperature 15-VR P1 Mem 2 Temperature: 27.0 °C
Temperature 16-VR P2 Mem 1 Temperature: 38.0 °C
Temperature 17-VR P2 Mem 2 Temperature: 39.0 °C
Temperature 18-Chipset Temperature: 35.0 °C
Temperature 19-BMC   Temperature: 47.0 °C
Temperature 20-HD Max Temperature: 40.0 °C
Temperature 22-Stor Batt Temperature: 22.0 °C
Temperature 23-E-Fuse Temperature: 38.0 °C
Temperature 24.1-HD Controller-Add-in card Temperature: 21.0 °C
Temperature 24.2-HD Controller-I/O controlle Temperature: 27.0 °C
Temperature 24.3-HD Controller-I/O controlle Temperature: 24.0 °C
Temperature 24.4-HD Controller-Add-in card Temperature: 22.0 °C
Temperature 25.1-HD Cntlr Zone-Add-in card Temperature: 21.0 °C
Temperature 25.2-HD Cntlr Zone-Add-in card Temperature: 22.0 °C
Temperature 26-I/O Zone 1 Temperature: 35.0 °C
Temperature 27-I/O Zone 2 Temperature: 37.0 °C
Temperature 28-I/O Zone 3 Temperature: 41.0 °C
Temperature 29-P1 Mem Zone 1 Temperature: 30.0 °C
Temperature 30-P1 Mem Zone 2 Temperature: 31.0 °C
Temperature 35.1-Mezz 3-I/O module Temperature: 59.0 °C
Temperature 35.2-Mezz 3-External Environment Temperature: 39.0 °C
Temperature 36.1-Mezz 3 Inlet-External Envir Temperature: 39.0 °C
Temperature 37-Sys Exhaust 1 Temperature: 32.0 °C
Temperature 38-Sys Exhaust 2 Temperature: 34.0 °C
Temperature 39.1-ExpBayBoot-I/O controller Temperature: 39.0 °C
Temperature 39.2-ExpBayBoot-I/O controller Temperature: 20.0 °C
Volume 1             Raid Type: RAID1, Size: 447.1GB
Volume 65            Raid Type: None, Size: 894.3GB
Volume 73            Raid Type: None, Size: 894.3GB
No piggyback files for '<<HOSTNAME>>'. Skip processing.
No piggyback files for '<<IP-ADDRESS>>'. Skip processing.
[cpu_tracking] Stop [7f3b9b0e83b0 - Snapshot(process=posix.times_result(user=4.93, system=0.03, children_user=0.0, children_system=0.0, elapsed=4.969999998807907))]
[special_redfish] Success, [piggyback] Success (but no data found for this host), execution time 8.2 sec | execution_time=8.200 user_time=4.930 system_time=0.040 children_user_time=0.360 children_system_time=0.040 cmk_time_ds=2.820 cmk_time_agent=0.000

andreas-doehler · February 3, 2025, 6:47am

This looks good 3.2 seconds for the real agent runtime.

Pay attention that you remove the debug option from the special agent settings. It writes a log file every time it is executed if not removed.

If you want to improve the runtime a little bit more, it is possible to define a cache time for the memory section. Every module state is not needed to fetch at every run. As you have the “Memory Summary” state automatically from the system section.
In my systems i let run the memory section with an cache time of 10 minutes. If one module has a problem then the summary check will switch to warning or critical and the corresponding module will also be in warning or critical latest 10 minutes (or set cache time) later.
Why this can help you would ask. Every module state needs to be fetched with one query and that takes time also if the normal answer time is fast.

redl_jan · February 12, 2025, 2:01pm

Thank’s!
Yes, I’ve disabled the debug option

The caching option makes indeed a big difference!

user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]

system · February 12, 2026, 2:02pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.