Upgrade 2.3.0p45 -> 2.4.0p24 results in errors on activation

[Glowsome (Michael Honkoop)]

CMK version: 2.3.0p45 CRE/community
OS version: Rocky9

Error message:
During Upgrade errors are given regarding the Redfish plugin:

Creating temporary filesystem /omd/sites/comsolve/tmp...OK
Executing 'cmk-update-config --conflict ask --dry-run'
-| ATTENTION
-|   Some steps may take a long time depending on your installation.
-|   Please be patient.
-|
-| Cleanup precompiled host and folder files
-| Verifying Checkmk configuration...
-| Error loading rulespecs: [ValueError("cmk_addons.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_datasource_programs: plug-in 'redfish' already defined at cmk.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_datasource_programs"), ValueError("cmk_addons.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_power_datasource_programs: plug-in 'redfish_power' already defined at cmk.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_power_datasource_programs"), ValueError("cmk_addons.plugins.redfish.rulesets.redfish_ethernetinterfaces:rule_spec_discovery_redfish_ethernetinterfaces: plug-in 'discovery_redfish_ethernetinterfaces' already defined at cmk.plugins.redfish.rulesets.redfish_ethernetinterfaces:rule_spec_discovery_redfish_ethernetinterfaces"), ValueError("cmk_addons.plugins.redfish.rulesets.redfish_outlets:rule_spec_discovery_redfish_outlets: plug-in 'discovery_redfish_outlets' already defined at cmk.plugins.redfish.rulesets.redfish_outlets:rule_spec_discovery_redfish_outlets")]
-| cmk_addons.plugins.redfish.graphing.translation:translation_redfish_outlets: plug-in 'redfish_outlets' already defined at cmk.plugins.redfish.graphing.translation:translation_redfish_outlets
-| cmk_addons.plugins.redfish.graphing.metrics:metric_input_power: plug-in 'input_power' already defined at cmk.plugins.redfish.graphing.power:metric_input_power
-| cmk_addons.plugins.redfish.graphing.metrics:metric_output_power: plug-in 'output_power' already defined at cmk.plugins.redfish.graphing.power:metric_output_power
-| cmk_addons.plugins.redfish.graphing.metrics:metric_input_voltage: plug-in 'input_voltage' already defined at cmk.plugins.redfish.graphing.voltage:metric_input_voltage
-| cmk_addons.plugins.redfish.graphing.metrics:metric_media_life_left: plug-in 'media_life_left' already defined at cmk.plugins.redfish.graphing.ssddrives:metric_media_life_left
-| cmk_addons.plugins.redfish.graphing.metrics:metric_ssd_utilization: plug-in 'ssd_utilization' already defined at cmk.plugins.redfish.graphing.ssddrives:metric_ssd_utilization
-| cmk_addons.plugins.redfish.graphing.perfometer:perfometer_input_output_power: plug-in 'power_summary' already defined at cmk.plugins.redfish.graphing.power:perfometer_input_output_power
-|  01/09 Legacy check plug-ins...
-|  02/09 Rulesets...
-|  03/09 UI extensions...
-| Error loading rulespecs: [ValueError("cmk_addons.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_datasource_programs: plug-in 'redfish' already defined at cmk.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_datasource_programs"), ValueError("cmk_addons.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_power_datasource_programs: plug-in 'redfish_power' already defined at cmk.plugins.redfish.rulesets.datasource_program:rule_spec_redfish_power_datasource_programs"), ValueError("cmk_addons.plugins.redfish.rulesets.redfish_ethernetinterfaces:rule_spec_discovery_redfish_ethernetinterfaces: plug-in 'discovery_redfish_ethernetinterfaces' already defined at cmk.plugins.redfish.rulesets.redfish_ethernetinterfaces:rule_spec_discovery_redfish_ethernetinterfaces"), ValueError("cmk_addons.plugins.redfish.rulesets.redfish_outlets:rule_spec_discovery_redfish_outlets: plug-in 'discovery_redfish_outlets' already defined at cmk.plugins.redfish.rulesets.redfish_outlets:rule_spec_discovery_redfish_outlets")]
-| cmk_addons.plugins.redfish.graphing.translation:translation_redfish_outlets: plug-in 'redfish_outlets' already defined at cmk.plugins.redfish.graphing.translation:translation_redfish_outlets
-| cmk_addons.plugins.redfish.graphing.metrics:metric_input_power: plug-in 'input_power' already defined at cmk.plugins.redfish.graphing.power:metric_input_power
-| cmk_addons.plugins.redfish.graphing.metrics:metric_output_power: plug-in 'output_power' already defined at cmk.plugins.redfish.graphing.power:metric_output_power
-| cmk_addons.plugins.redfish.graphing.metrics:metric_input_voltage: plug-in 'input_voltage' already defined at cmk.plugins.redfish.graphing.voltage:metric_input_voltage
-| cmk_addons.plugins.redfish.graphing.metrics:metric_media_life_left: plug-in 'media_life_left' already defined at cmk.plugins.redfish.graphing.ssddrives:metric_media_life_left
-| cmk_addons.plugins.redfish.graphing.metrics:metric_ssd_utilization: plug-in 'ssd_utilization' already defined at cmk.plugins.redfish.graphing.ssddrives:metric_ssd_utilization
-| cmk_addons.plugins.redfish.graphing.perfometer:perfometer_input_output_power: plug-in 'power_summary' already defined at cmk.plugins.redfish.graphing.power:perfometer_input_output_power
-| [redfish 2.3.76]: Ignoring problems (MKP will be disabled on target version)

This is not blocking the upgrade, however after the upgrade (which finishes as global status of OK, starting the site, trying to update the configuration of the site results in the following message:

If i check omd status , then the board is green.

I rolled back to a snapshot, retried it, but same result unfortunately.

I’m looking for directions as to troubleshoot this on a future attempt.

• Glowsome

[andreas-doehler (Andreas)]

The first messages only comes from the redfish mkp what is later in the update process disabled.

The UI scheduler is here more the problem. You can check the scheduler on command line also his log is important.

[Glowsome (Michael Honkoop)]

Well after alot of time where i had no chance to reproduce the upgrade and the error finally got time to redo it (2.3.0p46 → 2.4.0p27), and what i found in /omd/sites/Mysite/tmp/run/ is the following issue:

OMD[Mysite]:~/tmp/run$ ll
total 16
-rw-r--r-- 1 Mysite Mysite   4 May  7 00:35 agent-receiver.pid
-rw-r--r-- 1 Mysite Mysite   4 May  7 00:35 automation-helper.pid
srwx------ 1 Mysite Mysite   0 May  7 00:35 automation-helper.sock=
-rw------- 1 Mysite Mysite   4 May  7 00:35 cmk-ui-job-scheduler.pid
srw-rw---- 1 Mysite Mysite   0 May  7 00:35 live=
lrwxrwxrwx 1 Mysite Mysite   8 May  7 00:35 live-tcp -> live-tls
drwxr-x--x 2 Mysite Mysite 120 May  7 00:35 mkeventd/
prw-rw---- 1 Mysite Mysite   0 May  7 00:35 nagios.cmd|
srwx------ 1 Mysite Mysite   0 May  7 00:35 redis=
-rw------- 1 Mysite Mysite   4 May  7 00:35 redis-server.pid
srw-rw---- 1 Mysite Mysite   0 May  7 00:35 rrdcached.sock=
srw------- 1 Mysite Mysite   0 May  7 00:35 ui-job-scheduler.sock=

All sockets have a ‘=’ added, and nagios.cmd has a pipe-character added ?
This compared to a testbox (already on 2.4)

-rw-r--r-- 1 testsite testsite    4 May  6 23:26 agent-receiver.pid
-rw-r--r-- 1 testsite testsite    4 May  6 23:27 automation-helper.pid
srwx------ 1 testsite testsite    0 May  6 23:27 automation-helper.sock
-rw------- 1 testsite testsite    4 May  6 23:27 cmk-ui-job-scheduler.pid
srw-rw---- 1 testsite testsite    0 May  6 23:27 live
lrwxrwxrwx 1 testsite testsite    8 May  6 23:26 live-tcp -> live-tls
-rw------- 1 testsite testsite 2087 May  7 00:00 logrotate.state
drwxr-x--x 2 testsite testsite  120 May  6 23:26 mkeventd
prw-rw---- 1 testsite testsite    0 May  6 23:27 nagios.cmd
srwx------ 1 testsite testsite    0 May  6 23:26 redis
-rw------- 1 testsite testsite    4 May  6 23:26 redis-server.pid
srw-rw---- 1 testsite testsite    0 May  6 23:26 rrdcached.sock
srw------- 1 testsite testsite    0 May  6 23:27 ui-job-scheduler.sock

if i revert the site to a pre-upgrade snapshot i still see sockets with added ‘=’ and nagios.cmd with an added pipe.

-rw-r--r-- 1 Mysite Mysite   4 May  7 00:56 agent-receiver.pid
srw-rw---- 1 Mysite Mysite   0 May  7 01:06 live=
lrwxrwxrwx 1 Mysite Mysite   8 May  7 00:56 live-tcp -> live-tls
drwxr-x--x 2 Mysite Mysite 120 May  7 00:56 mkeventd/
prw-rw---- 1 Mysite Mysite   0 May  7 00:56 nagios.cmd|
srwx------ 1 Mysite Mysite   0 May  7 00:56 redis=
-rw------- 1 Mysite Mysite   4 May  7 00:56 redis-server.pid
srw-rw---- 1 Mysite Mysite   0 May  7 00:56 rrdcached.sock=

But on that 2.3.0p46 - version i have no issues …

By the looks of the error after upgrade its understandable that if it is looking for a socket without the added ‘=’.
So did the upgrade somehow forgot to correct this or ?
Also i have looked at service config files, and their corresponding service(init.d)-file but was unable to find where the added ‘=’ is coming from.

BTW: Both boxes are same OS, same patchlevel ( RockyLinux 9.7)

Update: i’ve done some digging, and for some reason the added characters seem to relate to previously upgraded OS-settings where a ‘=’ indicates a socket, and a pipe indicates (obviously) indicates a pipe.

So then i had a look at logs, where no mentions were there in regards of unable to start, or having other issues.
So my mystery remains.. before i had to rollback and resume normal operations on a/the attempted -to-upgrade machine.

As this is sort-of frustrating, as it is blocking my move up to the next major version (2.4), and potentially to the most current release (2.5.0) all help is welcome regarding this.

• Glowsome

[chauhan_sudhir]

@Glowsome

Also i have looked at service config files, and their corresponding service(init.d)-file but was unable to find where the added ‘=’ is coming from.

It comes from the alias:

OMD[v25pro]:~$ alias
alias cpan=‘cpan.wrapper’
alias la=‘ls -la’
alias ll=‘ls -l’
alias ls=‘ls --color=auto -F’

The = and | are not part of the filenames.They are added by ls -F / ll as file-type indicators:

= Unix socket
| FIFO / named pipe
/ directory

So:

live=
nagios.cmd|
redis=

actually means:

live is a socket
nagios.cmd is a named pipe
redis is a socket

You can verify with the below and then the suffixes should disappear:

type ll
type ls
command ls -l /omd/sites/Mysite/tmp/run
/bin/ls -l --indicator-style=none /omd/sites/Mysite/tmp/run

[Glowsome (Michael Honkoop)]

Thanks for the explanation,

However the issue blocking me to upgrade still exists, within the 2.3.0pX i can update without issue.
But moving from 2.3.0pX → 2.4.0pX is still producing same issue(s)

[chauhan_sudhir]

For ui-job-scheduler, I would recommend a grep for this omd service in the syslog and also dmesg and look for OOM:

cat /var/log/syslog|grep -i ui-job
cat /var/log/messages|grep -i ui-job

If the ui-job-scheduler is stopped on the site, then please check the following logs for any obvious errors:

$OMD_ROOT/var/log/ui-job-scheduler/error.log

$OMD_ROOT/var/log/ui-job-scheduler/ui-job-scheduler.log

$OMD_ROOT/var/log/web.log

If any custom MKP’s are broken during the upgrade, then try to do the update without it.

[Glowsome (Michael Honkoop)]

i have tried updating/upgrading again, and unfortunately the same issue still exists ( both 2.3.0pX and 2.4.0pX are latest available.

Taken actions this time:

• removed/disabled all external MKP’s prior to update/upgrade.

During Update/Upgrade:

• No errors and/or warnings are seen.

After Upgrade:

According to omd status all processes are running without issue.

@chauhan_sudhir

$OMD_ROOT/var/log/ui-job-scheduler/error.log

OMD[mysite]:~$ tail -f var/log/ui-job-scheduler/error.log
2026-05-22 00:57:07,060 [20] [8639/MainThread] Started server process [8639]
2026-05-22 00:57:07,061 [20] [8639/MainThread] Waiting for application startup.
2026-05-22 00:57:07,083 [20] [8639/MainThread] Application startup complete.
2026-05-22 00:57:07,085 [20] [8639/MainThread] Uvicorn running on unix socket /omd/sites/mysite/tmp/run/ui-job-scheduler.sock (Press CTRL+C to quit)

@chauhan_sudhir

$OMD_ROOT/var/log/ui-job-scheduler/ui-job-scheduler.log

OMD[mysite]:~$ tail -f var/log/ui-job-scheduler/ui-job-scheduler.log
2026-05-22 01:02:49,387 [20] [8639/MainThread] Process terminated (Exit code: 0)
2026-05-22 01:02:49,387 [20] [8639/MainThread] Stopping application
2026-05-22 01:02:49,388 [20] [8639/scheduler] Waiting for jobs to finish
2026-05-22 01:02:49,388 [20] [8639/scheduler] Stopped scheduler
2026-05-22 01:02:56,625 [20] [14070/MainThread] --- Starting ui-job-scheduler (Checkmk 2.4.0p29) ---
2026-05-22 01:02:59,957 [20] [14070/MainThread] Starting scheduler thread
2026-05-22 01:02:59,958 [20] [14070/scheduler] Started scheduler
2026-05-22 01:02:59,964 [20] [14070/MainThread] Starting server
2026-05-22 01:02:59,984 [20] [14070/MainThread] Starting background jobs on_scheduler_start hooks
2026-05-22 01:02:59,987 [20] [14070/MainThread] Finished on_scheduler_start hooks

@chauhan_sudhir

$OMD_ROOT/var/log/web.log

OMD[mysite]:~$ tail -f var/log/web.log
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/mysite/lib/python3/cmk/gui/watolib/activate_changes.py", line 1496, in start
    self._start_activation()
  File "/omd/sites/mysite/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/mysite/lib/python3/cmk/gui/watolib/activate_changes.py", line 1803, in _start_activation
    raise result.error
cmk.gui.job_scheduler_client.StartupError: Could not connect to ui-job-scheduler. Possibly the service <tt>ui-job-scheduler</tt> is not started, please make sure that all site services are all started. Tried to connect via <tt>/omd/sites/mysite/tmp/run/ui-job-scheduler.sock</tt>. Reported error was: HTTPConnectionPool(host='local-ui-job-scheduler', port=80): Max retries exceeded with url: /start (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x78cce02011c0>: Failed to establish a new connection: [Errno -2] Name or service not known')).

If i stat the /omd/sites/mysite/tmp/run/ui-job-scheduler.sock it does return a/the socket-file:

OMD[mysite]:~$ stat /omd/sites/mysite/tmp/run/ui-job-scheduler.sock
  File: /omd/sites/comsolve/tmp/run/ui-job-scheduler.sock
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 6fh/111d        Inode: 6121        Links: 1
Access: (0600/srw-------)  Uid: (  987/mysite)   Gid: ( 1004/mysite)
Access: 2026-05-22 01:02:59.987748340 +0200
Modify: 2026-05-22 01:02:59.987748340 +0200
Change: 2026-05-22 01:02:59.988748341 +0200
 Birth: 2026-05-22 01:02:59.987748340 +0200

So without errors - except the connection to the socket i am at a total loss

[chauhan_sudhir]

What’s the output of:

• omd status
  • If ui-job-scheduler shows as stopped, can you start it ? Does it stops again ?
• Are you using any proxy at the site level or in the Global settings? Can you disable this temporarily and see if the ui-job-scheduler works then?
• Please check for OOM’s in the OS syslog and dmesg.
• How much system resources you have ? Checkmk 2.4.0 requires slightly * higher system resources than 2.3.0. It is therefore advisable to find out before updating which free capacity the servers used for Checkmk still have.

[Glowsome (Michael Honkoop)]

@chauhan_sudhir

What’s the output of:

• omd status

As stated before everything is green/OK on the board

@chauhan_sudhir

Are you using any proxy at the site level or in the Global settings? Can you disable this temporarily and see if the ui-job-scheduler works then?

There is no proxy in use.

@chauhan_sudhir

Please check for OOM’s in the OS syslog and dmesg.

Did not see any.

@chauhan_sudhir

How much system resources you have ? Checkmk 2.4.0 requires slightly * higher system resources than 2.3.0. It is therefore advisable to find out before updating which free capacity the servers used for Checkmk still have.

More then enough, it has 8 cpu’s and i only see 28% of memory available in use.

[Glowsome (Michael Honkoop)]

With the help of a session with @chauhan_sudhir this issue has been resolved.